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Executive  Summary 

Over  the  past  decade,  the  United  States  Marine  Corps  (USMC)  has  shifted  its  training  focus  towards 
enabling  effective  and  efficient  decision-making  (DM)  in  its  small  unit  leaders.  Small  unit  leaders  are 
increasingly  required  to  make  decisions  with  both  tactical  and  strategic  impact  in  the  heat  of  the  battle. 
Simulation-based  training  provides  an  opportunity  for  trainees  to  consolidate  DM  skills  learned  in  the 
classroom  and  practice  making  decisions  in  stressful  environments  prior  to  entering  resource-intensive 
live  exercises.  No  collective  and  consolidated  guidance,  however,  is  available  on  how  to  utilize  simulation 
to  train  small  unit  leader  DM  under  stress.  Guidance  is  needed  in  several  key  areas,  including:  1)  merging 
standards-based  training  with  DM-based  training,  2)  inducing  stress  at  levels  that  impact  the  decision 
process  and  force  coping  strategies,  3)  integrating  objective  and  quantitative  assessments  to  understand 
performance  deficiencies  and  4)  incorporating  learning  strategies  that  can  enhance  DM  skills  and 
increase  resilience  to  stress. 

The  Small-unit  Training  for  Adaptability  and  Resilience  in  Decision-Making  (STAR-DM)  effort  described 
herein  addresses  these  needs  by  developing  a  simulation-based  training  approach  which  outlines 
methods  for  scenario  design,  stress  induction,  assessment,  and  learning  strategy  integration.  There  were 
three  main  goals  of  STAR-DM: 

1.  Develop  and  validate  a  generalizable  training  framework  (including  the  overall  model,  measures, 
and  learning  strategies)  to  better  train  adaptable,  stress-resilient  small  unit  leader  decision  makers 
in  simulation  environments, 

2.  Implement  the  training  framework  into  Marine  Corps-specific  simulation-based  Squad  Leader 
Training  Packages  (SLTPs)  and  validate  their  training  effectiveness,  and 

3.  Integrate  solutions  into  STAR-DM  assessment  and  debrief  tools  to  make  the  training  framework 
and  SLTPs  easily  accessible  to  instructors. 

In  order  to  develop  the  training  framework,  there  were  three  key  iterative  areas  of  focus;  field  research, 
theoretical  research  and  empirical  research.  The  field  research  focused  on  understanding  the  types  of 
stressful  decisions  that  Squad  Leaders  typically  face  as  well  as  identifying  gaps  in  current  training 
approaches.  This  research  revealed  user  needs  for  training  DM  under  stress,  as  well  as  the  content  for  a 
library  of  stressful  decision  events  faced  in  theater,  which  provided  a  basis  for  simulation  decision  events. 
Theoretical  research  was  conducted  to  understand  the  state  of  the  science  on  DM  under  stress, 
resilience,  measures,  and  learning  strategies,  and  an  overall  conceptual  model  for  training  DM  under 
stress.  This  research  resulted  in  the  development  of  a  training  framework  and  hypotheses  for  measures 
of  both  DM  and  stress,  ways  to  induce  stress,  and  learning  strategies  to  improve  both  DM  skills  and  ability 
to  cope  with  stress.  Empirical  research  then  looked  to  validate  the  training  framework  and  inform  the 
design  of  the  simulation  training  packages  and  assessment  and  debrief  tools. 

As  a  result  of  the  empirical  research,  four  SLTPs  were  designed  and  developed  and  vetted  at  the  Infantry 
Small  Unit  Leader  Course.  The  packages  are  Virtual  Battlespace  2  (VBS2)-based  packages  focused  on 
Squad  Leader  decision-making,  and  are  designed  to  be  turnkey,  high-throughput  exercises.  They  are 
designed  for  a  network  setting  in  which  a  full  USMC  squad  can  perform  a  squad  level  mission.  Each 
package  contains  four  VBS2  scenarios  with  a  running  narrative  and  take  approximately  a  half  to  a  full  day 
to  complete.  Each  SLTP  exposes  the  trainees  to  20  discrete  decision  events  and  has  an  integrated  pre¬ 
brief,  assessment,  and  paper-based  debrief  tools  along  with  intelligent  enemies  minimizing  the  need  for 
instructor  intervention.  A  tablet-based  software  tool  for  after  action  review  was  also  designed  and 
developed  to  support  instructors  in  real-time  assessment  of  performance  during  the  SLTPs. 

Based  on  experimental  findings,  the  STAR-DM  framework  and  SLTPs  were  shown  to  induce  a  significant 
physiological  stress  response,  even  in  experienced  Marines.  The  training  framework,  which  includes 
process-level  feedback  and  biofeedback,  led  to  improved  decision-making  performance  during  training,  a 
reduced  stress  response  in  trainees  and  showed  promise  for  increasing  overall  mission  performance. 
Results  also  suggest,  but  due  to  low  experimental  sample  size  do  not  definitely  prove,  that  the 
performance  improvements  resulting  from  the  use  of  STAR-DM  transfer  to  the  field,  leading  to  improved 
decision-making  performance  in  stressful  field  exercises.  Future  research  is  still  needed  in  terms  of  a 
training  effectiveness  evaluation  in  order  to  fully  evaluate  the  training  effectiveness  and  field  transfer  of 
the  complete  STAR-DM  training  framework. 
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1.0  Introduction 

Over  the  past  decade,  the  United  States  Marine  Corps  (USMC)  has  shifted  its  training  focus  towards 
enabling  effective  and  efficient  decision-making  (DM)  in  its  small  unit  leaders.  Small  unit  leaders  with 
relatively  little  experience  are  increasingly  required  to  make  tactical  decisions  with  critical  second  and 
third  order  effects.  These  near  strategic  level  decisions  are  not  being  made  in  a  Command  Operations 
Center  (COC),  but  in  the  heat  of  the  battle,  where  the  decision  maker  is  surrounded  by  high  levels  of 
physical  and  emotional  stress.  Key  to  the  development  of  effective  decision  makers  is  training,  which 
effectively  targets  decision-making  skills.  The  USMC  has  recognized  this  and  made  it  a  priority.  In  the 
2012  Marine  Corps  Science  and  Technology  (S&T)  Strategic  Plan,  a  key  Training  and  Education  (T&E) 
Science  and  Technology  Objective  (STO)  is  focused  specifically  on  Warrior  decision-making  (STO  1), 
calling  for  the  development  of  products  and  technologies  to  assist  Marines  at  all  levels  in  better  preparing 
to  make  effective  decisions  in  complex  environments.  Additionally,  T&E  STO  2  focuses  specifically  on 
small  unit  learning  and  performance  assessment,  requesting  evaluation  technologies  and  methodologies 
and  scenario-based  measures  to  enhance  feedback  and  after  action  review  (AAR).  Further,  studies  have 
shown  significant  adverse  effects  of  combat  stressors  on  cognitive  performance  (Lieberman  et  al.,  2005) 
as  well  as  persistent  changes  in  brain  functional  connectivity  (Van  Wingen  et  al.,  2012).  In  fact,  the  2012 
Marine  Corps  S&T  Strategic  Plan  also  includes  T&E  STO  3  for  Warrior  Resilience  and  Med  STO-9  for 
Stress  resistance,  resilience,  and  recovery,  calling  for  products  and  technologies  that  enhance  the 
understanding  of  and  the  training  for  resilience  to  stress.  T&E  STO  4  also  calls  for  experiential  learning 
technologies  and  methodologies  that  increase  the  capacity  and  quality  of  training  due  to  the  limited  time 
and  resources  available  for  training.  To  ensure  military  success,  and  the  health  and  wellness  of  our 
veterans,  it  is  critical  that  small  unit  leaders  receive  efficient  and  effective  training  necessary  to  develop 
strategies  which  enable  them  to  make  effective  decisions  under  stress  and  mitigate  long  term 
physiological  and  psychological  impacts  of  stress. 

The  USMC  36th  Commandant’s  Planning  Guidance  (2015)  calls  for  a  focus  on  better  leveraging 
simulation  training,  especially  capabilities  that  support  the  development  of  resilient  leaders  and  sound 
tactical  and  ethical  decision-making  at  the  small  unit  level.  It  states: 

“Our  investment  in  training  systems  will  reflect  the  priority  we  place  on  preparing  for  combat  and 
be  fully  integrated  with  training  and  readiness  standards.  I  expect  all  elements  of  the  MAGTF 
(Marine  Air-Ground  Task  Force)  to  make  extensive  use  of  simulators  where  appropriate.  My 
intent  is  for  Marines  to  encounter  their  initial  tactical  and  ethical  dilemmas  in  a  simulated 
battlefield  vice  actual  combat.”  (p.  11). 

Simulation-based  training  provides  an  opportunity  for  trainees  to  consolidate  DM  skills  learned  in  the 
classroom  (Cohn  et  al.,  2007)  and  to  practice  making  decisions  in  stressful  environments  (Cannon, 
Bowers  and  Salas,  1998)  prior  to  entering  resource-intensive  live  exercises.  Simulation-based  training 
also  provides  the  opportunity  to  expose  trainees  to  a  large  array  of  situations,  environments  and  decision 
points  not  possible  in  live  exercises  due  to  logistical  and  resource  constraints.  Such  experience  supports 
building  up  a  trainee’s  experience  base  -  a  necessity  in  moving  an  individual  from  a  novice,  analytic 
decision-maker  to  a  more  expert,  recognition-based  decision-maker  (Marine  Corps  Institute,  2010).  No 
collective  and  consolidated  guidance,  however,  is  available  on  how  to  utilize  simulation  to  train  small  unit 
leader  decision-making  under  stress.  Guidance  is  needed  in  several  key  areas  to  ensure  effective 
simulation  based  training,  including:  1)  merging  standards-based  training  (e.g.,  Training  and  Readiness 
(T&R)  standards)  with  decision-making-based  training,  2)  inducing  stress  at  levels  that  impact  the 
decision  process  and  force  coping  strategies,  3)  integrating  objective  and  quantitative  assessment  to 
understand  performance  deficiencies  and  4)  incorporation  of  learning  strategies  that  can  enhance 
decision-making  skills  and  increase  resilience  to  stress  (USMC,  2012). 

The  Small-unit  Training  for  Adaptability  and  Resilience  in  Decision  Making  (STAR-DM)  effort  described 
herein  addresses  these  needs  by  developing  a  simulation  based  training  approach  which  outlines 
methods  for  scenario  design,  stress  induction,  assessment,  and  learning  strategy  integration.  There  were 
three  main  goals  of  STAR-DM: 
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1.  Develop  and  validate  a  generalizable  STAR-DM  training  framework  (including  the  overall  model, 
measures,  and  learning  strategies)  to  better  train  adaptable,  stress-resilient  small  unit  leader 
decision  makers  in  simulation  environments, 

2.  Implement  the  training  framework  into  Marine  Corps-specific  simulation-based  Squad  Leader 
Training  Packages  (SLTPs)  and  validate  their  training  effectiveness,  and 

3.  Integrate  solutions  into  STAR-DM  assessment  and  debrief  tools  to  make  the  training  framework 
and  SLTPs  easily  accessible  to  instructors. 

Table  1  below  provides  a  summary  of  current  approaches  and  how  STAR-DM  improves  upon  those 
approaches. 


Table  1.  Current  Training  vs.  STAR-DM  Approach 


Training 

Framework 

Model, 

Metrics, 

Learning 

Strategies 


Simulation 

Training 


Assessment 
and  Debrief 
Tools 


How  is  it  done  today? 
Limitations  of  current  practice? 


Stress/DM  targeted  separately; 
Interdependence  not  considered. 

Metrics  at  outcome  level;  not  granular 
enough  to  identify  root  cause. 

No  guidance  for  learning  strategies  to 
target  decision  making  under  stress. 

Unstructured  scenarios  focused  on 
T&R  events  lack  ability  to  effectively 
train  decision  making. 

Consumes  valuable  training  time  with 
limited  exposure  to  decision 
experiences. 

Assessment  subjective,  outcome  level; 
limited  by  instructor  recall  and 
workload. 

Debrief  based  on  notes/salient 
memories;  at  times  imprecise  and 
incomplete. 


What’s  new  in  the  STAR-DM  approach? 
Why  will  it  succeed? 


Unified  DM  under  stress  model,  measures 
and  training  techniques. 

Objective,  quantitative,  granular,  metrics 
(behavioral,  physiological)  to  tailor  training. 

Integrated,  tailored  learning  strategies  to 
bolster  resilience  of  DM  process. 

Merges  standards-based  with  DM  training 
via  scenarios  designed  to  elicit  adaptive  DM 
skills. 

Leverages  the  power  of  simulation  to  quickly 
build  novice  decision  maker’s  experience 
base. 

Objective  assessment  tools  free  up  instructor 
cognitive  resources  to  monitor  performance. 

Debrief  tools  provide  snapshot  of  where/why 
breakdowns  occurred  so  debrief  can  be 
tailored  to  specific  breakdowns  in  need  of 
remediation. 


2.0  Technical  Approach 

The  technical  approach  for  STAR-DM  followed  an  iterative  design  and  development  approach  as 
indicated  in  Figure  1.  The  first  step  focused  on  identification  of  domain  requirements,  wherein  the  team 
worked  closely  with  USMC  Infantry  Small  Unit  Leader  Course  (ISULC)  instructors  to  understand  small 
unit  leader  decision-making  and  the  impact  of  stress  on  those  decisions.  Next,  theoretical  research  was 
conducted  to  identify  the  state  of  the  science  in  effective  DM  training  and  assessment.  An  initial  training 
framework  was  developed,  and  a  series  of  empirical  research  evaluations,  laboratory  studies,  and  a  field 
study  with  Marines  were  conducted  to  validate  the  framework  and  fill  theoretical  gaps.  The  results  of 
these  studies  drove  the  development  of  requirements  for  the  framework  design,  as  well  as  design  of  the 
simulation  training  packages  and  assessment  and  debrief  tools.  The  effort  was  supposed  to  end  with  a 
culminating  training  effectiveness  evaluation  to  assess  the  impact  of  the  STAR-DM  training  framework 
and  tools.  Unfortunately,  as  will  be  discussed  later,  that  experiment  did  not  take  place.  The  following 
sections  will  discuss,  in  more  detail,  each  of  the  steps  of  the  technical  approach. 
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Figure  1.  STAR-DM  Technical  Approach 


2.1  Field  Research  Overview 

In  order  to  determine  how  best  to  fill  training  gaps  in  small  unit  leader  decision-making  under  stress,  it 
was  first  necessary  to  understand  the  current  training  given  to  small  unit  leaders  in  the  Marine  Corps. 
Therefore,  the  first  step  of  the  technical  approach  was  to  conduct  field  research  in  the  form  of  a  task 
analysis  of  small  unit  decision-making  at  the  USMC  ISULC  at  the  School  of  Infantry  East  (SOI-E).  The 
research  team  was  looking  to  define  the  type  of  stressful  decisions  that  small  unit  leaders  need  to  make 
and  how  they  are  trained  to  make  them.  While  at  ISULC  SOI-E,  the  research  team  also  conducted  a 
training  needs  analysis  to  identify  training  gaps  that  could  be  addressed  with  the  simulation  training 
framework  being  developed. 

The  ISULC,  developed  by  the  United  States  Marine  Corps,  is  a  six  week  course  directed  specifically  at 
Infantry  Sergeants  with  Squad  Leader  experience.  The  goal  of  the  course  is  to  develop  leadership  and 
DM  capabilities  by  exercising  critical  thinking,  supporting  cognitive  development  and  challenging  a 
leader’s  ability  to  solve  problems  [USMC  AITB  (Advanced  Infantry  Training  Battalion)  New  Course  Brief, 
2012],  Whereas  traditional  Programs  of  Instruction  (POIs)  focus  on  specific  technical  and  tactical  skills, 
the  purpose  of  ISULC  is  to  propel  experienced  Sergeants  (E-5  with  Squad  Leader  experience)  to  high 
decision-making  expertise  levels  by  creating  opportunities  for  the  trainees  to  leverage  and  add  to  their 
previously  gained  experience  and  technical  and  tactical  knowledge  to  more  effectively  assess  situations 
and  make  decisions.  Thus,  ISULC  is  a  POI  focused  on  application  -  specifically  to  develop  expert 
decision  makers  who  can  excel  at  leading  small  units  in  combat.  To  achieve  this  end,  numerous  training 
techniques  and  materials  are  employed,  including  traditional  classroom  lecture,  computer-based 
simulation,  and  live  training.  All  ISULC  training  is  rooted  in  T&R  Manual  tasks  with  a  heavy  focus  on 
guided  discussions,  decision  forcing  cases,  tactical  decision  games  (TDGs),  sand-table  exercises 
(STEXs),  and  simulation-based  training  (USMC,  2011).  This  garrison  training  feeds  into  several  field 
exercises  and  live-fire  training  ranges  where  the  Sergeants  rotate  through  a  variety  of  billets  from  Platoon 
Commander  to  Squad  Leader,  providing  them  the  opportunity  to  make  a  variety  of  decisions  across  a 
large  range  of  situations. 
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In  order  to  identify  critical  small  unit  leader  decisions  to  include  in  the  STAR-DM  training  framework, 
specific  decision  points  were  extracted  from  a  variety  of  ISULC  training  events  (e.g.  classroom  training, 
STEXs,  field  exercises),  which  were  observed  by  the  research  team.  Furthermore,  the  research  team  sat 
down  with  seven  Marines,  with  a  variety  of  Squad  Leader  experience,  to  extract  decisions  they  faced  in 
theater  under  stress.  This  comprehensive  list  of  relevant  small-unit  leader  decisions  was  then  categorized 
into  a  library  of  decisions  to  be  used  in  the  development  of  the  training  scenarios  (Appendix  B). 

2.2  Theoretical  Research  Overview 

The  STAR-DM  effort  sought  to  develop  a  training  methodology  that  is  not  only  operationally  relevant,  but 
also  theoretically  sound.  Therefore,  in  parallel  to  the  field  research  conducted  at  ISULC  SOI-E,  the 
research  team  also  reviewed  the  literature  for  models,  measures,  and  training  techniques  related  to 
decision-making,  stress,  resilience,  and  adaptability  (see  Hannigan  et  al.,  2012,  for  details).  Models  of  the 
DM  process  were  reviewed,  and  the  OODA  loop  (Observe,  Orient,  Decide,  Act;  Boyd,  1987)  was  chosen 
as  the  model  to  base  the  STAR-DM  effort  off  of  due  to  its  general  acceptance  in  the  military.  The  effects 
of  stress  on  decision-making  were  then  reviewed,  specifically  on  each  step  of  the  OODA  loop,  as  well  as 
on  the  overall  process.  The  literature  review  also  sought  to  identify  measures  of  both  DM  and  stress  and 
ways  to  induce  stress  to  include  in  the  empirical  research  portion  of  the  STAR-DM  effort.  Finally,  the 
review  sought  to  identify  learning  strategies  to  improve  both  DM  and  an  individual’s  ability  to  recognize 
stress  overload  and  cope  with  it,  enabling  the  individual  to  focus  on  sound  decision-making. 

2.3  Empirical  Research  Overview 

Once  the  theoretical  gaps  were  defined,  the  next  step  in  the  technical  approach  was  for  the  STAR-DM 
research  team  to  design  four  experiments  with  the  aim  of  addressing  the  gaps  in  the  theory.  Experiment 
1,  the  first  lab  experiment,  was  designed  to  address  three  questions: 

•  Can  simulation  scenarios  induce  stress,  and  if  so,  how  does  it  compare  to  the  gold  standard 
socio-evaluative  stressor? 

•  Which  physiological  measures  are  most  indicative  of  stress  and  resilience? 

•  Which  DM  measures  are  most  discriminative/predictive? 

The  findings  from  Experiment  1  were  that  stress  can  be  induced,  measured,  and  it  does  have  an  impact 
on  DM  skills.  Given  this  information,  Experiment  2  (i.e.  second  lab  experiment)  was  designed  to  address 
one  additional  question: 

•  Can  learning  strategies  (i.e.  process  level  feedback  and  biofeedback)  effectively  improve 
decision-making  under  stress  in  infantry-based  simulation  scenarios? 

The  outcome  of  Experiment  2  indicated  that  DM  learning  strategies  and  process  level  feedback  improve 
DM  performance  and  stress  strategies  and  biofeedback  reduce  the  stress  response.  Furthermore,  both 
process-level  feedback  and  biofeedback  showed  promise  for  increasing  overall  mission  performance. 

While  Experiments  1  and  2  focused  on  the  general  population,  the  Field  Study  and  the  planned,  but  not 
executed,  training  effectiveness  evaluation  (TEE)  focused  on  the  target  population  of  Marine  small  unit 
leaders.  The  field  study  was  designed  to  determine  if  the  effects  of  the  Experiment  1  would  translate  to 
the  Marine  population.  The  TEE  was  designed  to  investigate  if  the  learning  strategies  from  Experiment  2 
would  demonstrate  the  same  effects  on  the  target  Marine  population  and  whether  the  training  efficacy  of 
the  full  STAR-DM  training  framework  would  transfer  to  the  field.  The  outcome  from  the  field  experiment 
showed  that  a  stress  response  can  be  induced  in  Marines  using  simulation-based  training  and  that  the 
training  can  improve  DM  performance.  Additionally,  the  improved  performance  appears  to  transfer  to  the 
field,  though  due  to  low  sample  size  further  study  is  needed.  The  TEE  was  not  executed,  therefore  the 
effect  of  learning  strategies  on  Marines  and  training  transfer  of  the  complete  STAR-DM  package  was  not 
able  to  be  assessed.  More  details  about  all  four  studies  including  the  methods  and  results  are  found  in 
Section  3.3. 
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2.4  Simulation  Squad  Leader  Training  Packages  (SLTP)  Overview 

One  of  the  steps  of  the  technical  approach  was  to  design  four  simulation  packages  based  on  the  findings 
from  the  field,  theoretical,  and  empirical  research.  The  packages  were  designed  for  ISULC  SOI-E  (SLTPs 
1  &  2)  and  for  the  Combating  Terrorism  Technical  Support  Office  (CTTSO)  (SLTPs  3  &  4).  SLTPs  1  &  2 
were  initially  created  to  support  data  collection  in  the  field  study  and  were  refined  based  on  the  results 
from  the  field  study  and  Experiment  2. 

The  packages  are  Virtual  Battlespace  2  (VBS2)-based  packages  focused  on  Squad  Leader  DM,  and  are 
designed  to  be  turnkey,  high-throughput  exercises.  The  packages  are  designed  for  a  network  setting  in 
which  a  full  USMC  squad  can  perform  a  squad  level  mission.  Each  package  contains  four  VBS2 
scenarios  with  a  running  narrative  and  take  approximately  a  half  to  a  full  day  to  complete.  Each  package 
exposes  the  trainees  to  20  discrete  decision  events  and  has  an  integrated  pre-brief,  assessment,  and 
debrief  tools  along  with  intelligent  enemies  to  minimize  the  requirement  for  instructor  intervention. 

The  training  packages  are  comprised  of  the  following  components: 

1.  Multimedia  Videos:  VBS2-based  Road  to  War  and  Operational  Update  videos  detail  the  situation 
and  environment  and  are  designed  to  engage  and  motivate  the  trainees. 

2.  Mission  Orders/  Fragmentary  Orders  (FRAGOs):  Paper-based  and/or  video-based  mission 
orders/FRAGOs  are  included  for  delivery  by  the  instructor/facilitator  for  viewing  by  the  trainees. 
After  receiving  mission  orders,  the  packages  are  designed  to  allow  the  Squad  Leader  to  plan  the 
mission  and  task  fire  teams  based  on  these  orders.  There  are  guidelines  provided  to  make  this  a 
more  stressful  experience,  such  as  time  pressure  and  instructor  maintaining  eye  contact  and 
providing  pointed  feedback. 

3.  VBS2  Scenarios:  Each  package  contains  four  VBS2  scenarios  designed  to  allow  the  entire  squad 
to  execute  the  Squad  Leader’s  plan  in  a  networked  VBS2  simulation.  As  the  team  performs  the 
scenario,  they  encounter  five  discrete  decision  events  in  which  the  Squad  Leader  has  to 
determine  the  best  course  of  action  and/or  adapt  his  plan. 

4.  Measurement  and  Debrief  Tools:  The  packages  have  integrated  paper-based  assessment  tools 
that  allow  instructors  to  assess  performance  at  the  process  level.  The  assessment  tools  are 
designed  to  also  facilitate  a  DM-focused  debrief. 

5.  Instructor  Preparation  Guides  (IPGs):  Each  package  is  accompanied  by  an  IPG,  which  walks  an 
instructor  step-by-step  through  delivery  of  the  training  packages  and  contains  supporting 
materials  such  as  maps,  orders,  and  performance/debrief  checklists. 

Additional  details  about  the  SLTPs  can  be  found  in  Section  4. 

2.5  Assessment  and  Debrief  Tool  Overview 

The  assessment  and  debrief  tool  was  developed  to  address  the  training  gap  in  which  instructors  required 
objective  measures  by  which  to  assess  a  Squad  Leader's  decision-making.  The  first  version  of  the  tool 
was  a  paper-based  instructor  checklist  with  a  breakdown  of  all  the  Squad  Leader's  expected  OODA  items 
per  event,  both  good  and  bad.  The  OODA  items  were  ranked  from  best  to  worst,  and  space  was  left  for 
any  alternative  OODA  items  chosen  by  the  Squad  Leader  that  were  not  present  on  the  form.  The 
checklist  is  arranged  in  such  a  way  that  the  instructor  can  easily  collect  and  scan  information  about  the 
Squad  Leader's  DM  process,  and  use  the  form  to  facilitate  a  process-level  debrief  regarding  the  strengths 
and  weaknesses  of  the  Squad  Leader's  decision-making.  The  paper-based  assessment  checklists  were 
used  in  the  field  study  by  Marine  instructors  and  refined  based  on  their  feedback.  The  strong  upside  to 
this  version  is  that  it  requires  no  extra  technology  to  implement.  The  downside  is  that  the  instructor  is 
either  tasked  with,  or  unable  to  keep  track  of,  minute  simulation  details  such  as  the  event  number  or  the 
number  of  times  the  squad  is  hit  by  enemy  fire. 
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The  second  version  of  the  assessment  checklist  is  a  software  tool  that  has  the  event  checklist 
programmed  in,  is  connected  with  the  VBS  simulation  to  monitor  simulation  details  such  as  the  event 
number  and  a  variety  of  performance-based  metrics,  and  has  a  checklist  editor  so  that  the  instructor  can 
make  edits  or  create  new  checklist  assessments.  The  software  version  of  the  Assessment  and  Debrief 
tool  is  designed  to  allow  the  instructor  to  focus  on  the  Squad  Leader's  DM  process  and  reference  metrics 
when  appropriate  to  facilitate  the  Squad  Leader's  training.  The  downside  is  that  the  software  version 
requires  an  additional  computer  or  tablet.  The  computer  can  weigh  the  instructor  down  from  moving  about 
the  classroom  to  facilitate  situational  awareness  of  the  events  in  the  simulation.  The  use  of  a  tablet  solves 
this  issue,  but  faces  the  hurdles  of  needing  wireless  connectivity  that  is  difficult  to  get  approved  for  use  in 
the  Marine  Corp  because  of  security  concerns.  Details  regarding  the  assessment  and  debrief  can  be 
found  in  Section  5. 

2.6  Final  Training  Effectiveness  Evaluation  (TEE)  Overview 

In  order  to  determine  if  the  potential  gains  demonstrated  in  the  laboratory  and  field  studies  would  transfer 
to  the  real  world  environment,  a  training  effectiveness  evaluation,  or  TEE,  was  designed.  This  experiment 
had  the  additional  goals  of  evaluating  the  effectiveness  and  return  on  training  investment  of  other  training 
environments.  The  primary  research  question  for  the  study  was:  How  does  the  training  effectiveness  of 
the  following  training  environments  compare:  Classroom  training  (in  the  form  of  TDGs),  Simulation 
training  (STAR-DM  SLTPs),  training  on  a  basic  non-instrumented  range  (such  as  62  Area  RUF  at  Camp 
Pendleton),  training  on  a  more-sophisticated,  instrumented  range  (such  as  Kilo2  at  Camp  Pendleton),  and 
training  at  Camp  Pendleton’s  Infantry  Immersion  Trainer  (NT). Though  a  plan  for  the  study  was  created 
and  significant  effort  was  expended  in  an  attempt  to  secure  a  unit  to  participate  in  the  testing,  the  team 
and  program  sponsors  were  unable  to  find  the  necessary  available  participants  to  execute  the  study.  An 
overview  of  the  planned  TEE  is  discussed  in  Section  3.3.4. 


3.0  STAR-DM  Training  Framework 

The  overall  objective  of  the  STAR-DM  framework  is  to  provide  operationally  relevant  guidance  that  is 
based  on  a  strong  foundation  of  theoretical  and  empirical  research  on  how  to  train  infantry  small  unit 
leaders  to  make  effective  decisions  under  stress.  The  framework  includes  not  only  the  overall 
methodology,  but  also  tools  in  the  form  of  the  SLTPs,  which  include  an  overall  process  that  induces 
stress,  simulation  scenarios  that  provide  specific  opportunities  for  small  unit  leaders  to  practice  decision¬ 
making,  and  associated  assessment  tools  and  directions  for  instructors.  The  SLTPs  are  meant  to  be 
turnkey  training  packages  that  provide  1-2  days  of  training  with  little  preparation  needed  from  the 
instructors. 

In  order  to  develop  the  training  framework,  there  were  three  key  iterative  areas  of  focus;  field  research, 
theoretical  research  and  empirical  research.  The  field  research  focused  on  understanding  the  types  of 
stressful  decision  events  that  Squad  Leaders  typically  face  as  well  as  identifying  gaps  in  current  training 
approaches.  Theoretical  research  was  conducted  to  understand  state  of  the  science  from  the  research 
reviewed  on  decision-making  under  stress,  resilience,  measures,  and  learning  strategies,  an  overall 
conceptual  model  for  training  decision-making  under  stress.  Empirical  research  then  looked  to  validate 
the  training  framework  and  inform  the  design  of  the  simulation  training  packages  and  assessment  and 
debrief  tools.  More  details  on  the  work  conducted  in  each  of  these  areas  in  presented  in  the  following 
sections. 

3.1  Field  Research:  Understanding  Small  Unit  Decision  Making  Under  Stress 

The  field  research  was  conducted  at  ISULC  SOI-E.  It  consisted  of  two  primary  parts;  a  task  analysis  to 
understand  the  types  of  decisions  a  Squad  Leader  would  typically  encounter  and  a  training  needs 
analysis  to  understand  gaps  in  current  decision-making  training. 
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3.1.1  Task  Analysis 

Goal 

The  main  goal  was  to  identify  types  of  decision  events  that  Squad  Leaders  are  often  presented  with  in 
order  to  create  a  library  from  which  decisions  could  be  selected  to  embed  in  VBS2  scenarios.  This  was 
achieved  through  two  means.  First,  portions  of  the  course  were  observed  to  extract  decisions  events 
used  in  practical  application  exercises  as  well  as  those  presented  anecdotally  to  trainees  by  instructors. 
Second,  stressful  decisions  experienced  by  Small  Unit  leaders  when  deployed  in  theater  and  acting  as 
Squad  Leaders  were  extracted  through  subject  matter  expert  (SME)  interviews.  A  secondary  goal  was  to 
extract  additional  information  to  better  understand  how  the  decisions  were  made  and  how  to  assess 
them.  Thus,  information  regarding  decision  considerations,  prerequisite  knowledge  needed,  situation 
assessment,  and  decision  outcomes  was  sought. 

Methods 

Two  primary  data  collections  methods  were  utilized  to  extract  decisions  and  associated  considerations; 
course  observation  and  interviews.  Course  observations  were  conducted  at  ISULC  during  the  organic 
weapons  portion  of  the  course.  Thirteen  Squad  Leaders  participated  in  this  instantiation  of  the  ISULC. 
Two  researchers  attended  both  morning  and  afternoon  portions  of  the  class  over  the  observation  period. 
Over  that  time  period,  instruction  included  instructor-led  lecture  using  PowerPoint  slides,  demonstration  of 
equipment,  and  practice  in  writing  tasking  orders  and  mission  plans  using  sand  table  exercises. 

During  the  sand  table  exercises,  researchers  observed  and  recorded  the  types  of  decisions  and  the 
considerations  informing  those  decisions.  These  were  extracted  from  discussion  following  the  trainees' 
briefs,  and  discussion  between  the  instructor  and  the  group  of  trainees  in  which  the  instructor  would  ask  if 
they  considered  aspects  of  the  situation  or  what  they  would  do  if  certain  events  occurred.  Stress  factors 
were  also  extracted  from  these  discussions. 

Researchers  also  interviewed  nine  Marines  with  Squad  Leader  experience.  However,  two  of  interviewees 
were  unable  to  provide  the  sought  after  information  because  of  limited  experience  as  Squad  Leaders  in 
theater.  Therefore,  data  collected  from  seven  interviews  provided  relevant  decision  events  presented 
herein.  The  seven  interviewees  ranged  in  age  from  24-39  (average  of  31)  years  and  ranged  in  rank  from 
Lance  Corporal  to  Gunnery  Sergeant.  Their  average  number  of  deployments  was  five  and  average  time 
as  a  Squad  Leader  was  approximately  two  years,  not  including  the  Gunnery  Sergeant  who  had  been  a 
Squad  Leader  for  only  three  months  before  acting  as  a  platoon  sergeant  for  eight  years.  Each  interview 
lasted  approximately  one  hour  and  followed  the  following  format: 

1 .  Each  interviewee  was  instructed  on  the  purpose  of  the  interview  and  the  potential  benefits  that 
would  come  from  the  interviews. 

2.  Demographic  data  was  collected  including  age,  rank,  military  occupational  specialty,  number  of 
deployments,  location  of  deployments,  and  amount  of  time  spent  as  a  Squad  Leader. 

3.  The  SMEs  were  then  asked  to  describe  situations  where  they  had  to  make  difficult  and  stressful 
decisions  as  a  Squad  Leader,  including  how  they  decided  what  to  do  and  what  were  alternative 
courses  of  action.  They  were  told  that  the  researchers  would  not  be  judging  the  decisions  made 
in  any  way,  and  that  they  were  merely  trying  to  collect  data  on  operationally  relevant  situations  in 
which  stressful  decisions  were  made. 

4.  The  interviewees  then  described  decisions  made  in  theater  when  acting  as  a  Squad  Leader  that 
were  stressful,  where  several  decision  alternatives  existed,  and  where  they  may  or  may  not  have 
selected  the  most  optimal  course  of  action. 

5.  The  researcher  then  asked  the  interviewee  a  series  of  amplifying  questions  regarding  the 
decision. 


Results 

The  methods  resulted  in  the  identification  of  37  decision  events  (see  Appendix  B)  and  associated 
information  to  include  the  following: 

a)  Background  information:  General  situation,  task/mission  objective  and  decision  descriptions. 
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b)  Prerequisite  knowledge:  Intelligence,  Commander’s  intent,  previous  experience  in  the  area, 
enemy  tactics,  techniques,  and  procedures  (TTPs),  etc.  that  factored  into  the  decision. 

c)  Data  collection:  Multimodal  cues  observed  which  impacted  the  decision. 

d)  Situation  assessment:  Situational  factors  and  understanding  of  the  situation  based  on  a 
combination  of  the  prerequisite  knowledge  and  cues  observed. 

e)  Courses  of  action:  Alternative  courses  of  action  identified  by  the  decision-maker  as  options  based 
on  the  situation  assessment. 

f)  Action  taken:  Course  of  action  taken  by  the  decision-maker. 

g)  Decision  outcome:  Outcome  of  the  decision  based  on  the  action  taken  by  the  decision-maker,  as 
well  as  potential  outcomes  of  alternative  courses  of  actions. 

h)  Stress  factors:  Environmental  and  task  factors  that  made  the  decision  more  stressful  for  the 
decision-maker. 

These  decision  events  were  then  categorized  in  an  attempt  to  roll  the  events  up  into  higher  level  decision 
areas.  The  events  fell  into  five  primary  decision  area  categories: 

1 .  Act  autonomously  versus  act  with  support 

2.  How  to  move  in  danger  area 

3.  Shoot/no  shoot 

4.  Abort  mission  versus  continue 

5.  Type  of  method  to  employ  to  achieve  mission  objective 

3.1.2  Training  Needs  Analysis 

Instruction  at  ISULC  SOI-E  occurs  via  a  combination  of  lecture,  guided  discussion,  TDGs,  STEXs, 
simulation,  and  field  exercises.  It  was  determined  that  the  ISULC  would  benefit  from  a  number  of 
developmental  enhancements  in  each  of  the  observed  sections  [sim,  live-fire,  finish  of  exercise  (FINEX)] 
across  the  areas  of  training  objectives,  training  content,  performance  measurement,  and  interventions. 
Table  2  highlights  how  each  of  these  areas  can  be  improved  upon.  These  recommendations  guided  the 
development  of  the  SLTPs  and  were  presented  to  ISULC  as  opportunities  to  enhance  the  training 
effectiveness  of  the  ISULC  curriculum. 
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Table  2:  Opportunities  for  Increasing  Effectiveness  of  Decision  Making  Assessment  and  Training 


Opportunities 

Training 

Objectives 

The  benefits  of  VBS2  can  be  better  utilized  by  focusing  the  training  on  the  execution  of 
missions  in  addition  to  mission  planning.  Simulation  facilitates  the  insertion  of  a  large 
array  of  scenario  cues  to  which  the  trainee  must  observe,  orient,  decide  and  act.  This 
provides  the  flexibility  to  expose  the  trainees  to  a  large  number  of  varying  environments 
and  situations.  Specific  training  objectives  can  be  attained  through  the  use  of  VBS2  by 
carefully  structuring  scenarios  with  the  inclusion  of  multimodal  cues  designed  to  elicit 
and  test  decision  making  processes  at  varying  difficulty  levels.  Additionally,  scenarios 
can  be  built  to  provide  positive  or  negative  reinforcement  to  actions  taken  by  the  unit  by 
allowing  them  to  see  the  consequences  of  their  decisions  play  out. 

Recommendations: 

•  Focus  training  on  decision  making  during  both  the  planning  of  missions  as  well  as 
execution  to  provide  trainees  the  opportunity  to  see  the  pros  and  cons  of  their  plan 
as  it  plays  out  and  make  dynamic  decisions  necessary  to  adapt  their  plan  to  the 
changing  situation. 

•  Utilize  VBS2  scenarios  systematically  structured  via  integration  of  training  objective- 
based  scenario  decision  events  designed  to  elicit  adaptive  decision  making  skills. 

Training 

Content 

ISULC  covers  numerous  decision  areas  throughout  the  course.  Simulation  and  live 
training  exercises  can  be  used  to  consolidate  the  skills  learned  during  the  other  ISULC 
training  modules.  To  achieve  this,  it  is  necessary  to  identify  ISULC  target  decision  area 
training  objectives  and  design  simulation  and  live  training  exercises  to  allow  trainees  to 
gain  experience  making  these  types  of  decisions  in  realistic  environments  that  provide 
powerful  feedback  as  the  effects  of  the  decisions  play  out.  Further,  designing  the 
scenarios  such  that  they  take  less  time  to  plan  and  execute,  more  trainees  would  have 
the  opportunity  to  act  as  the  squad  leader  and  gain  experience  making  decisions. 

Recommendations: 

•  Identify  ISULC  target  decision  areas  and  create  Master  Scenario  Event  Lists 
(MSELS)  for  all  simulation  and  live  training  exercises  that  map  scenario  events  to 
these  decision  areas. 

•  Reduce  length  of  scenarios  to  allow  a  greater  number  of  missions  executed  per  day. 

Performance 

Measurement 

Current  performance  measures  utilized  in  simulation  and  live  exercises  are  both 
subjective  and  at  the  outcome  level.  This  provides  limited  diagnostics  to  facilitate 
targeting  feedback  to  the  root  cause  of  performance  deficiencies  (for  instance,  was  the 
problem  that  the  squad  leader  wasn’t  utilizing  the  right  information  or  did  he  not  select  a 
good  decision  alternative?).  Instructors  need  tools  that  are  easy  to  use  and  allow  them 
to  objectively  assess  decision  making  at  the  process  level.  The  Decision  Making 
Assessment  Tool  (DM AT)  is  a  good  starting  point,  but  the  mixed  reactions  of  the 
instructors,  their  reports  of  areas  for  improvement,  and  their  general  lack  of  complete 
satisfaction  with  the  DMAT  indicate  room  for  refinement.  There  is  a  clear  need  for 
reliable,  valid  assessment  measures  that  can  be  used  to  more  comprehensively 
evaluate  each  student. 

Recommendations: 

•  Simulation:  Utilize  VBS2  system-collected  measures  to  capture  objective,  process 
level  decision  making  performance  measures. 

•  Live:  Utilize  a  combination  of  event-based  performance  checklists  to  capture 
objective,  process  level  decision  making  performance  measures  in  hand  with 
pre/post-test  measures  such  as  Situational  Judgment  Tests  (SJTs). 

-  Systematically  gather  feedback  from  ISULC  instructors  regarding  the  DMAT  and 
refine  and  validate  improved  assessment  tool. 

-  Identify,  develop,  and  field  test  additional  decision  making  assessment  measures 
such  as  SJTs. 
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Interventions 


Within  the  current  format  of  the  simulation  portion  of  the  ISULC,  there  is  a  significant 
amount  of  time  for  the  instructors  to  provide  feedback  and  other  training  interventions. 
While  peer-to-peer  and  instructor  feedback  on  the  outcome  is  important,  interventions 
could  be  more  robust  and  targeted.  For  instance,  maintaining  or  building  an  alignment 
between  the  current  training  and  what  students  already  know  as  with  Advance 
Organizers,  or  critically  examining  a  mission  brief  and  the  anticipated  mission  during 
crystal  ball  exercises  or  pre-mortem  exercises  can  help  build  Marines’  decision-making 
skills.  What  is  needed  are  pre,  during  and  post  training  techniques  and  tools  which 
allow  trainees  to  become  fully  immersed  in  the  training,  practice  dynamic  decision 
making  skills  and  receive  real-time  and  after  action  feedback  targeted  to  their  specific 
skill  deficiencies. 


Recommendation: 


•  Incorporate  into  simulation  training  pre,  during,  and  post  training  interventions  which 
have  been  shown  to  enhance  decision-making  skills  and  are  easy  for  instructors  to 
deliver. 


Based  on  observation  and  later  confirmed  by  the  AITB  ISULC  website,  the  class  is  approximately  75% 
practical  application  and  25%  classroom  instruction.  These  practical  application  exercises,  especially  the 
computer-based  simulation  exercises,  provide  a  prime  opportunity  for  utilization  of  the  STAR-DM  training 
framework  presented  above  as  they  provide  an  opportunity  for: 

1)  Instantiation  of  stressors  and  numerous  environmental  variations, 

2)  Measurement  of  trainee  performance  and  stress  response,  and 

3)  Incorporation  of  learning  strategies  such  as  metacognitive  feedback  and  biofeedback  methods. 

In  addition  to  the  above  opportunities  and  recommendations,  there  are  three  key  limitations  or  gaps  in  the 
computer-based  simulation  exercises  which  the  STAR-DM  framework  can  also  help  address,  First,  the 
simulation  exercises  lack  a  deliberate  approach  to  scenario  design  that  targets  the  DM  process  and 
allows  for  the  trainees  to  hone  underlying  decision  skills  (e.g.,  OODA).  Further,  during  the  observed 
computer  simulation-based  training  there  was  a  lack  of  emphasis  on  specific  decision  themes 
(commander’s  intent,  METT-TC  (Mission,  Enemy,  Terrain  &  Weather,  Troops,  Time  Available,  and 
Civilian  Considerations)  analysis,  Terrain  Analysis,  Battlespace  Geometry,  etc.)  that  were  a  focus  during 
the  TDGs  and  guided  discussions  conducted  during  the  first  weeks  of  the  ISULC  course.  Scenarios  are 
needed  which  allow  the  trainees  to  consolidate  the  decision  skills  learned  in  the  classroom  and  develop 
skills  and  coping  strategies  to  a  level  that  is  resilient  to  stress. 

A  second  limitation  consistently  observed  throughout  all  practical  applications  is  the  lack  of  instantiated 
metrics  with  which  to  assess  the  DM  process  of  the  Sergeants.  In  order  to  effectively  train  stress  resilient 
decision  skills,  it  is  necessary  to  measure  the  decision  process  as  well  as  stress  response.  Without  these 
measures  in  place,  it  is  not  possible  to  tailor  the  training  to  the  decrements  of  the  trainees  and  facilitate 
the  development  of  decision  skills,  and  stress  appraisal  and  coping  skills  required  to  make  decisions 
under  stress. 

Finally,  decision-focused  learning  strategies  are  utilized  throughout  the  course.  However,  they  did  not 
focus  on  building  resilience  to  stress. 

STAR-DM  developed  Instructor  Preparation  Guides  (IPGs)  with  guidance  aimed  at  addressing  the  above 
mentioned  gaps  and  improving  the  ISULC  ability  to  target  small  unit  DM  under  stress.  These  IPGs 
integrate  decision  targeted  scenarios  with  metrics  and  learning  strategies  aimed  to  build  stress  resilient 
DM  skills.  The  scenarios  and  IPGs  are  further  discussed  in  Section  4. 

3.2  Theoretical  Research  and  STAR-DM  Conceptual  Model 

Theoretical  research  was  conducted  in  three  primary  areas;  decision-making  and  stress,  resilience  to 
stress  and  learning  strategies.  Information  gleaned  from  the  state  of  the  science  review  of  those  areas 
informed  the  STAR-DM  conceptual  model  which  formed  the  basis  for  the  STAR-DM  training  approach 


14 


□ESIGNinteractive 


Design  Interactive,  Inc. 


Contract#:  N00014-12-G-0427 


and  SLTP  development.  The  following  sections  provide  a  summary  of  the  research  performed  and  the 
conceptual  model. 

3.2.1  Decision  Making  and  Stress 

In  order  to  elucidate  the  effects  of  stress  on  DM,  it  was  first  necessary  to  define  the  DM  process.  Many 
models  of  decision-making  detail  a  four  step  process.  Models  such  as  the  OODA  Loop  (Observe,  Orient, 
Decide,  Act;  Boyd,  1987),  the  SHOR  model  (Stimulus,  Hypothesis,  Options,  Response;  Wohl,  1981),  and 
the  CASE  model  (Collect  data,  Assess  situation,  Select  response,  Evaluate  response;  Johnston  et  al., 
1998),  decompose  the  DM  process  into  these  core  processes:  1)  collection  and  integration  of  sensory 
information,  2)  interpretation  of  this  sensory  information  to  determine  the  current  environment  and 
situation,  3)  evaluation  of  alternative  courses  of  actions  and  response  selection,  and  4)  planning  and 
execution  of  the  response.  For  the  STAR-DM  effort,  the  OODA  Loop  (Figure  2)  was  chosen  as  the  DM 
model  of  focus  because  it  has  traditionally  been  used  by  military  commanders  as  a  descriptive  framework 
of  the  military  DM  process  and  therefore  provides  operational  relevance  to  the  conceptual  model. 


Observe 


Act  Orient 


Decide 


Figure  2.  OODA  Loop  (Boyd,  1987) 

Stress  can  be  described  as  occurring  when  a  person  appraises  an  environment  as  taxing  or  exceeding 
available  resources  and  endangering  personal  well-being  (Lazarus  and  Folkman,  1984).  According  to 
Stokes  and  Kite  (2001)  stress  may  be  viewed  as  “...an  agent,  circumstance,  situation  that  disturbs  the 
‘normal’  functioning  of  the  individual...  stress  is  seen  as  an  effect  -  that  is  the  disturbed  state  itself ...  this 
difference  in  meaning  is  arguably  the  most  fundamental  source  of  the  confusion  surrounding  the  stress 
concept.”  (p.  109).  In  their  review,  Stokes  and  Kite  (2001)  asserted  that  there  are  two  traditional  models  of 
psychological  stress:  a  stimulus-based  and  a  response-based.  The  stimulus-based  stress  model  assumes 
that  certain  conditions  cause  stress  and  labels  these  conditions  stressors.  For  example,  any  dismounted 
patrol  in  an  unfamiliar  area  can  be  a  stressor  if  one  appraises  it  as  taxing.  Other  examples  of  stressors 
may  include  heat  and  cold,  time  pressure,  etc.  The  authors  argued  that  the  stimulus-based  approach  is 
inadequate  to  explain  stress  and  the  human  response  because  it  does  not  explain  individual  differences, 
ignores  emotion,  and  does  not  evaluate  circumstances  (Staal,  2004).  The  response-based  stress 
approach  proposed  that  stress  is  defined  by  the  pattern  of  responses  (i.e.,  physiological,  cognitive,  and 
affective)  that  are  caused  by  a  given  stressor.  This  approach  focuses  on  the  individual’s  reaction  instead 
of  the  mechanical  nature  of  stress  and  response  (the  stimulus-based  model)  of  a  given  stressor.  A  third 
model  of  stress  called  the  transactional  model  takes  a  different  look  at  stress.  Instead  of  viewing  stress  as 
a  stimulus  or  an  individual  response,  it  focuses  on  the  interaction  between  the  environment  and  the 
individual  (Stokes  and  Kite,  2001).  It  emphasizes  the  role  of  the  individual’s  appraisal  of  situations  in 
shaping  their  responses.  This  model  is  an  important  piece  in  the  present  work  associated  with  small  unit 
DM  because  it  provides  a  foundation  for  developing  training  strategies  to  build  individual  resilience.  By 
accounting  for  individual  stress  appraisal  differences,  training  strategies  can  target  individual  users. 
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Stressors  of  various  types  (physical,  psychological)  can  negatively  influence  the  DM  process,  leading  to 
physiological  changes  (e.g.,  increased  cortisol,  adrenaline,  and  serotonin)  as  well  as  cognitive  changes, 
such  as  decreased  cognitive  capacity  and  reasoning  (McNeil  and  Morgan,  2010). 

These  effects  may  be  seen  at  individual  stages  in  the  DM  process.  The  first  stage  of  the  DM  process, 
observation/data  collection,  requires  appropriate  and  timely  attention  allocation  to  task  relevant  cues. 
Stress  such  as  time  pressure,  workload,  and  anxiety  have  been  shown  to  lead  to  attentional  narrowing  by 
reducing  cue  utilization,  shrinking  the  perceptive  field,  and  reducing  an  individual’s  environmental  scan 
(Staal,  2004).  For  instance,  Entin  and  Serfaty  (1990)  found  a  reduction  in  the  frequency  and  amount  of 
information  sought  by  decision  makers  under  high-stress  conditions.  Further,  stress  creates  distracting 
psychological  (e.g.,  anxiety)  and  physiological  (e.g.,  increased  heart  rate)  responses,  which  can  draw 
attention  away  from  task  relevant  information  (Baradell  &  Klein,  1993).  Stress  can  also  result  in  reversion 
of  automated  performance  to  conscious  control,  wherein  attentional  resources  are  consumed  by  step-by- 
step  monitoring  of  task  performance  (Wickens  &  Hollands,  2000).  The  next  two  steps  in  the  DM  process, 
situation  assessment  and  decision  alternative  evaluation,  rely  heavily  on  performers’  working  memory 
(Endsley,  1995).  Stressors  such  as  anxiety,  noise,  fatigue,  extreme  temperature  and  military  combat 
significantly  reduce  working  memory  capacity  and  performance  of  working  memory  tasks,  thereby  limiting 
situation  assessment  (Staal,  2004).  This  reduced  working  memory  capacity  can  also  negatively  impact 
the  process  of  evaluating  and  selecting  decision  alternatives,  leading  to  a  reduction  in  the  number  and 
quality  of  alternatives  considered  (Staal,  2004).  Keinan  (1987,)  among  others,  showed  that  participants 
under  stress  are  affected  by  premature  closure,  in  which  “a  decision  is  reached  before  all  available 
alternatives  have  been  considered.”  Additionally,  those  participants  can  also  succumb  to  “nonsystematic 
scanning”  in  which  a  stressed  decision  maker  frantically  searches  for  a  solution  to  the  problem  (Keinan, 
1987).  In  a  stress  free  environment,  decision  makers  rely  on  a  number  of  heuristics  ranging  from  simple 
to  complex  (Gigerenzer  and  Selten,  2001)  to  select  the  best  option  to  meet  their  goals.  To  determine  the 
best  approach,  they  search  their  memories  for  the  criterion-linked  probability  information.  According  to 
Gigerenzer,  Haffrage,  and  Kleinbolting  (1991),  this  search  for  information  is  quick,  streamlined,  and  tends 
to  rely  on  the  most  valid  probabilistic  cue  separating  alternatives.  When  under  stress,  DM  tends  to 
become  rigid  with  fewer  alternatives  scanned  (Broder,  2003).  In  addition,  research  has  shown  that 
individuals  may  rely  more  on  previous  responses  regardless  of  previous  success  with  such  responses 
(Lehner  et  al.,  1997).  Finally,  the  execution  stage  has  also  been  shown  to  be  influenced  by  stress. 
Decreased  execution  can  come  in  many  forms,  such  as  increased  errors  and  movement  variability  on 
perceptual-motor  tracking  tasks  (van  Galen  &  van  Huygevoort,  2000)  and  increased  errors  in  heading, 
steering,  and  reduced  perceptual  sensitivity  on  driving  tasks  (Matthews  &  Desmond,  2002). 

In  general,  when  people  are  under  stress  the  DM  process  is  impaired,  leading  to  reductions  in  quality  of 
and  confidence  in  decisions.  These  findings  have  been  demonstrated  across  a  variety  of  domains  ranging 
from  firefighting  (Ozel,  2001)  to  aviation  (Wickens,  Stokes,  Barnett,  &  Flyman,  1991).  Thus,  the  challenge 
is  to  train  Warfighters  to  be  resilient  to  stress  by  enabling  the  development  of  strategies  to  counter  the 
physiological  and  psychological  impacts  of  stress  and  maintain  performance  while  completing  missions 
under  high  stress  conditions. 

3.2.2  Resilience  to  Stress 

Resilience  reflects  one’s  ability  to  maintain  stable  equilibrium  when  confronted  with  stress  or  endure  when 
presented  with  significant  challenges  (Bonnano,  2004;  Masten  &  Narayan,  2012).  Upon  exposure  to 
stress,  resilient  individuals  are  able  to  maintain  focus  by  appropriately  appraising  the  stressor(s)  and 
implementing  both  physiological  and  psychological  coping  strategies  as  necessary,  allowing  for  effective 
DM  skills  regardless  of  stressor(s)  present  (Lazarus,  1966).  Such  individuals  are  able  to  avoid  negative 
consequences  of  stress,  and  show  minimal  disturbance  to  performance  -  some  individuals  are  even  able 
to  thrive  under  such  conditions,  showing  greater  self-confidence  and  skills  (Epel  et  al.,  1998).  These 
individuals  have  been  shown  to  exhibit  strengths  in  stress  appraisal  (i.e.,  assessing  best/worst/likely 
outcomes  to  put  stress  into  perspective).  Studies  have  shown  that  resilience  can  be  improved  through 
various  methods,  especially  when  interventions  are  early  in  skill  development  (Feder,  Nestler,  and 
Charney,  2009).  For  example,  intermittent,  acute  stress  exposure  can  have  positive  effects  on  physiology, 
allowing  individuals  to  bounce-back  and  maintain  tight  allostasis  (the  process  of  maintaining  a  state  of 
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homeostasis;  Epel  et  al.,  1998).  Thus,  training  Warfighter  resilience  during  DM  training  should  focus  on 
two  main  objectives  1)  enhancing  adaptability  through  stress  appraisal  skills  and  2)  developing 
physiological  and  psychological  coping  strategies  that  support  recovery  (i.e.,  bounce-back)  and  effective 
performance  despite  the  experience  of  stress. 

Experimental  reports  on  this  subject  have  focused  on  long-term  trajectories  (weeks  to  months)  of 
resilience,  with  little  or  no  focus  on  whether  significant  changes  to  resilience  could  be  achieved  by  short¬ 
term  interventions  (Winslow  et  al.,  2013).  This  effort  proposed  that  the  quantification  of  an  individual’s 
physiological  and  behavioral  response  to  stress  under  controlled  conditions  is  an  indication  of  the 
individual’s  level  of  resilience,  meaning  that  short-term  interventions  could  lead  to  improvements  in 
resilience.  A  goal  of  the  training  framework  was  to  facilitate  assessment  of  a  trainee’s  resilience  to 
stressors  during  DM  performance  in  order  to  adapt  training  to  push  the  trainee  up  the  expertise 
continuum.  In  this  model,  it  was  proposed  that  adaptability  is  a  component  of  a  performer’s  resilience. 
Specifically,  resilience  can  be  modeled  as  a  function  of  1)  the  time  needed  to  recognize  that  the 
environment  has  changed  and  the  need  to  change  action,  which  aligns  with  the  initial  performance  drop 
resulting  from  a  stressor  -  the  adaptability  phase  and  2)  the  time  to  consider  decision  alternatives  and  act 
-the  bounce-back  phase.  See  Winslow  et  al.  (2015)  for  a  more  detailed  review  of  the  resilience  literature 
and  Carroll  et  al.  (2012)  for  a  more  detailed  explanation  of  the  model  for  training  resilient  decision¬ 
making. 

3.2.3  Assessment  of  Decision  Making  Performance  and  Stress 

Based  on  the  theoretical  research  on  resilience,  initial  efforts  in  determining  the  assessments  to  be 
utilized  in  the  STAR-DM  framework  focused  on  measures  of  adaptability  and  bounce-back.  Table  3  below 
describes  some  of  the  high-level  measures  for  each  of  these  constructs  (for  more  details  see  Carroll  et 
al.,  2012). 


Table  3.  Measures  of  Adaptability  and  Bounce-back 


Adaptability 

Bounce-back 

Recognition:  Ability  to  recognize  new  threats/ 
opportunities  and  orient  to  them  as  if  ample  time  to 
plan  and  prepare 

Perspective:  Ability  to  maintain  focus  and  change 
thoughts/behaviors  in  a  positive  manner  as  needed 
to  complete  tasks  despite  presence  of  stress 

Robustness:  Ability  to  degrade  gracefully  under 
attack  or  as  a  result  of  partial  failure 

Recoverability:  Ability  to  recover  from  decreased 
performance  as  a  result  of  exposure  to  stressors 

Stress  Appraisal:  Ability  to  accurately  assess  best 
case,  worst  case  and  most  likely  case  as  a  result  of 
stress  exposure 

Flexibility:  Ability  to  maintain/regain  effectiveness 
across  a  range  of  tasks,  situations,  and  conditions 

Allostatic  Load:  Ability  to  maintain  balance 
physiologically  in  the  presence  of  stress 

Allostatic  Load:  Ability  to  maintain  balance 
physiologically  in  the  presence  of  stress 

Agility:  Ability  to  recognize  when  to  shift  from  one 
strategy  to  another 

However,  as  more  work  was  done  on  the  design  of  the  conceptual  framework  and  implementation  of  the 
framework  for  training  DM  under  stress,  it  was  determined  that  measurement  needed  to  be  more  detailed 
and  should  really  focus  on  the  two  main  components  of  the  training  -  decision-making  and  stress  (Carroll 
et  al.,  2013). 

Decision-making  Assessment 

The  framework  for  assessing  DM  performance  is  based  on  the  four  stages  of  the  OODA  loop  (Boyd, 
1987).  Decomposition  of  the  DM  process  into  these  four  stages  allows  decision-making  to  be  assessed  at 
the  process  level  in  order  to  understand  the  root  cause  of  decision  breakdowns.  In  addition  to  typical 
decision  outcome  measures  (e.g.  decision  effectiveness  and  time  to  decide),  each  stage  of  the  decision 
process  can  be  assessed  independently.  For  the  Observe  stage,  there  are  critical  Areas  Of  Interest 
(AOIs)  which  should  be  attended  to  in  order  to  effectively  understand  the  situation.  Measures  for  the 
Observe  stage  could  include  eye  tracking  measures  such  as  percent  of  critical  AOIs  fixated  on  and 
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percent  of  critical  AOIs  fixated  for  greater  than  300  ms  (i.e.,  significant  attention  allocation,  see  Carroll, 
2010),  number  of  fixations,  and  average  and  total  fixation  duration  on  critical  AOIs  (for  metric  details,  see 
Carroll,  Kokini  and  Moss,  2013).  However,  during  the  testing  of  the  SLTPs,  these  measures  proved  to  be 
practically  infeasible.  Therefore,  the  Observe  phase  measures  ended  up  being  collected  similarly  to  those 
for  the  Orient  and  Decide  phases.  These  stages  are  much  more  challenging  to  assess  as  these  are 
generally  internal  and  unobservable  processes.  A  combination  of  two  techniques  is  used  to  assess  these 
stages  of  the  DM  process.  First,  contextually  relevant  online  queries  (Klein  &  Hoffman,  1992),  which  in 
the  military  context  can  be  instantiated  as  an  information  “pull”  from  higher  command,  are  incorporated 
into  the  scenario  in  which  decision  makers  are  prompted  to  answer  questions  related  to  the  events 
unfolding  as  they  perform.  Second,  by  crafting  scenarios  designed  to  elicit  certain  responses,  inference- 
based  measures  (Gugerty,  2011)  can  be  utilized  to  infer  the  Observe,  Orient  and  Decide  stages  by 
monitoring  communications  and  actions.  For  instance,  in  order  to  assess  the  Observe  and  Orient  stages, 
at  various  points  throughout  the  scenario  the  trainee  is  asked  to  verbally  report  what  they  see  and  what  it 
means.  Based  on  the  percentage  of  critical  situation  factors  reported,  observation  and  orientation 
performance  is  assessed.  In  order  to  assess  the  Decide  stage,  the  trainee  is  asked  to  verbally  report  the 
possible  courses  of  action  they  can  take.  Based  on  the  effectiveness  of  the  courses  of  action  reported 
and  the  actions  taken,  the  effectiveness  of  the  Decide  stage  is  assessed.  The  Action  phase  is  assessed 
based  on  how  effectively  the  response  is  carried  out.  However,  Action  is  not  a  major  focus  of  this  effort  as 
it  is  typically  an  indication  of  tactical/technical  skills,  not  decision  effectiveness.  These  measures  facilitate 
the  assessment  of  each  stage  independently  and  provide  granularity  in  assessment  to  understand  where 
breakdowns  in  the  DM  process  originated  and  to  identify  patterns  of  these  breakdowns  across  decisions 
(e.g.,  does  a  trainee  have  issues  with  observation  across  multiple  decisions  that  need  to  be  remediated). 

Stress  Assessment 

Stress  can  impair  the  DM  process  leading  to  reductions  in  quality  of  and  confidence  in  decisions  (for 
review  of  impacts  of  stress  on  decision-making  see  Carroll  et  al.,  2012).  Assessment  of  an  individual’s 
stress  response  during  DM  performance  can  help  to  determine  if  performance  decrements  are  due  to 
basic  performance  deficiencies  or  failure  to  be  resilient  to  stress,  and  therefore  provide  insight  into  how  to 
remediate  performance  decrements.  Stress  response  can  be  objectively  and  quantitatively  assessed,  in 
near  real  time,  without  disruption  to  task  performance  by  utilizing  physiological  measures.  When  an 
individual  encounters  a  significant  stressor,  the  body  experiences  hormone  release  and  subsequent 
physiological  effects  on  multiple  organ  systems,  including  heart  rate  increase,  pupil  dilation,  blood  vessel 
constriction,  and  activation  of  sweat  glands.  Physiological  data  representative  of  these  changes,  such  as 
electrocardiogram  (ECG),  electromyogram  (EMG),  electrodermal  activity  (EDA),  and  pulse 
plethysmography  (PPG)  can  be  measured  with  sensors  (Carroll  et  al.,  2013).  Measures  include:  1)  heart 
rate  variability  (HRV),  2)  pulse  transit  time  (PTT),  which  is  the  time  it  takes  the  pulse  waveform  to 
propagate  from  the  heart  to  the  periphery  (i.e.,  the  thumb),  3)  respiratory  sinus  arrhythmia  (RSA),  which  is 
a  naturally  occurring  variation  in  heart  rate  that  occurs  during  a  breathing  cycle,  4)  EMG  root  mean 
square  to  capture  neck  and  back  tension,  and  5)  electrodermal  response  (EDR),  spikes  which  occur  1  -  3 
seconds  following  stimulation  and  are  due  to  autonomic  innervation.  The  goal  was  to  identify  the  most 
robust  physiological  indicators  of  stress  response  in  order  to  develop  a  classification  method  for 
identifying  periods  during  training  when  an  individual  is  experiencing  significant  levels  of  stress. 

3.2.4  Learning  Strategies 

Individualized  learning  strategies  can  focus  on  building  resilience  into  the  DM  process  by  focusing  on  1) 
enhancing  stress  appraisal  skills,  particularly  in  early  stages  of  the  DM  process,  and  2)  developing 
physiological  and  psychological  coping  strategies  that  support  recovery  (i.e.,  bounce-back)  to  effective 
performance  levels.  Therefore,  similar  to  the  initial  review  of  assessment  strategies,  the  initial  review  of 
learning  strategies  also  focused  on  those  pertaining  to  adaptability  and  bounce-back.  A  few  of  these  are 
highlighted  in  Table  4  below  (see  Carroll  et  al.,  2012  for  more  details). 
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Table  4.  Learning  Strategies  for  Adaptability  and  Bounce-back 


Adaptability 

Bounce-back 

Metacognition  Strategies:  Guide  reflective  thought 
to  identify  best  case,  worst  case,  and  most  likely 
case  that  may  result  from  stressor  to  put  into 
perspective 

Stress  Inoculation  Training:  Conceptualization  and 
education  regarding  stressors,  skill  acquisition 
and  rehearsal  under  stress,  and  encouraging 
application  of  coping  skills 

Mindfulness  Training:  to  increase  attentional  control 
and  concentration  under  stress 

Biofeedback:  Provide  physiological  biofeedback  to 
increase  awareness  of  stress  response  and  guide 
conscious  evaluation  and  control  over 
physiological  state 

However,  the  challenge  is  identifying  the  optimal  strategies  that  are  effective  as  well  as  feasible  to 
implement  in  a  military  schoolhouse  setting.  Therefore,  this  set  of  learning  strategies  was  refined  based 
on  operational  constraints  while  expanding  it  to  include  learning  strategies  that  also  target  the  decision¬ 
making  process.  These  strategies  are  presented  in  Table  5  (from  Carroll  et  al.,  2013)  along  with  the 
associated  training  phase  (i.e.,  pre,  during,  post). 


Table  5.  Summary  of  Learning  Strategies 


Target  Construct 

Decision-making 

Stress  Response 

Adaptability 

Pre  Training 

Pre  Training 

Pre  Training 

Planning  Exercises 

Crystal  Ball  Exercises 

Bio  Feedback 

Motivation  Strategies 

Premortem  Exercises 

Durms  Training 

Dunne  Training 

Advanced  Organizers 

Stress  Exposure  Training  (SET) 

Emotional  Engagement 

/Stress  Inoculation  Training  (SIT) 

Vanety /Increasing  Complexity 

Dunns  Training 

Post  Training 

Novelty'.  Unexpected  Challenges 

Event-based  Training 

High  fidelity  Scenarios 

Metacognition  Strategies 

Post  Training 

Post  Training 

Error-based  Feedback 

Causal-based  feedback  \ 

Attribute  Isolation 

Mastery  Orientation 

Decision-making  Learning  Strategies 

There  are  several  learning  strategies  which  can  be  used  to  enhance  DM  skills  (Carroll  et  al.,  2013).  Pre¬ 
training  exercises  include:  planning  exercises  which  require  trainees  to  layout  the  plan  for  how  they  will 
accomplish  a  goal  have  been  effectively  utilized  (Blackburn  et  al.,  2004),  crystal  ball  exercises  in  which  a 
plan  is  scrutinized  for  potential  failure  points  by  assuming  that  a  ‘crystal  ball’  indicates  the  current 
assessment  is  wrong  and  it  is  necessary  to  explain  why  (Cohen  et  al.,  1998),  premortem  exercises  which 
assume  a  plan  has  failed  and  require  the  trainee  to  identify  plausible  reasons  for  the  failure  in  order  to 
critique  the  plan  (Veinott  et  al.,  2010),  and  Advanced  Organizers,  such  as  outlines,  narratives,  and  audio 
or  video  multimedia,  which  assist  learners  in  attending  to  the  most  appropriate  stimuli  and  effectively 
interpreting  information  during  training  (Vogel-Walcutt  et  al.,  2010;  Mueller-Hanson  et  al.,  2005).  A  key 
learning  strategy  which  can  be  utilized  during  training  is  an  Event-Based  Approach  to  Training  (EBAT; 
Fowlkes  et  al.,  1998)  in  which  training  opportunities  are  systematically  created  by  presenting  events 
designed  to  elicit  specific  skills  or  behaviors  such  as  decision-making.  After  action  review  techniques 
such  as  error-based  or  causal-based  feedback  (Carroll  et  al.,  2008,  Carroll,  2010)  provide  the  opportunity 
to  target  decision-making  at  the  process  level.  Errors  are  considered  to  be  valuable  opportunities  to  clarify 
misunderstandings  in  learners  (Mory,  2004)  and  the  identification  of  underlying  causes  of  deficiencies 
allows  instructors  to  provide  meaningful  feedback  (Salas  et  al.,  2007).  This  can  be  combined  with  specific 
learning  strategies  such  as  attribute  isolation  in  which  central  attributes  of  target  concepts  (e.g.,  why  cues 
are  critical  for  a  situations  or  indicative  of  an  impending  threat)  are  highlighted  to  improve  general 
understanding  of  phenomenon  (Mason  &  Bruning,  2001). 
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Stress  Response  Learning  Strategies 

There  are  also  various  learning  strategies  that  can  be  used  to  target  an  individual’s  stress  response 
during  DM  performance  (Carroll  et  al.,  2013).  Biofeedback  is  a  process  by  which  individuals  learn  to 
control  functions  such  as  blood  pressure,  salivation,  sweat  gland  activity,  and  cardiac  activity  through 
feedback  signals  from  sensors  monitoring  physiological  responses  (Calderon  &  Thompson,  2004).  Stress 
exposure  strategies  such  as  Stress  Inoculation  Training  (SIT;  Saunders  et  al.,  1996)  and  Stress  Exposure 
Training  (SET;  Driskell  &  Johnston,  1998)  have  also  been  shown  effective  in  increasing  a  trainee’s  ability 
to  cope  with  stress  (Sheehy  &  Horan,  2004;  Saunders  et  al.,  1996;  Meichenbaum  &  Deffenbacher,  1988). 
Metacognition  strategies  can  be  utilized  to  facilitate  awareness  of  how  one  perceives  and  thinks  about 
stressors  as  well  as  enable  accurate  appraisal  of  a  stressor  by  assessing  best,  worst,  and  most  likely 
outcomes  (Narayanan,  2009). 

Adaptability  Learning  Strategies 

Multiple  learning  strategies  can  be  used  to  increase  the  adaptability  of  a  decision  maker  (Carroll  et  al., 
2013).  Pre-training  strategies  include  techniques  to  increase  a  trainee’s  motivation  to  learn,  active 
engagement  in  training,  and  recognition  of  relevancy  to  their  job  (Mueller-Hanson  et  al.,  2009).  There  are 
also  several  learning  strategies  which  can  be  applied  during  simulation  training  exercises,  including 
variety  (e.g.,  variation  of  situations,  decisions),  increasing  difficulty/complexity  over  time,  novel  and 
unexpected  challenges  presented  to  stretch  the  trainee,  and  the  use  of  high  fidelity  scenarios  that  closely 
mimic  the  types  of  situations  they  will  encounter  in  the  field  (Mueller-Hanson  et  al.,  2009).  Post  training 
learning  strategies  such  as  mastery  orientation  drive  feedback  focused  on  the  process  not  the  outcome, 
and  can  also  be  utilized  to  increase  adaptability. 

From  the  research  reviewed  on  decision-making  under  stress,  resilience,  measures,  and  learning 
strategies,  an  overall  conceptual  model  for  training  decision-making  under  stress  was  developed  and  is 
described  in  the  section  below. 

3.2.5  STAR-DM  Conceptual  Model 

Within  small  unit  leaders,  the  ability  to  demonstrate  resilience  in  the  face  of  a  variety  of  stressors  is  critical 
to  mission  success.  Therefore,  training  Warfighter  resilience  during  decision-making  should  focus  on  two 
main  objectives  1)  enhancing  adaptability  through  stress  appraisal  skills  and  2)  developing  physiological 
and  psychological  coping  strategies  that  support  recovery  (i.e.,  bounce-back)  and  effective  performance 
despite  the  experience  of  stress.  Based  on  these  two  objectives,  a  framework  for  training  resilience  in 
decision-making  was  proposed  during  the  early  stages  of  this  effort,  based  on  the  pathways  model  of 
resilience  (adapted  from  Carver,  1998;  Figure  3).  Resilience  research  has  shown  that  when  a  person 
encounters  a  significantly  stressful  event,  he/she  will  either  1)  succumb  to  the  stress  and  performance  will 
degrade  to  the  point  of  failure,  2)  have  degraded  performance,  but  survive  by  maintaining  performance 
levels  that  enable  them  to  continue  to  operate  sub-optimally,  3)  recover  to  pre-stressor  levels,  or  4)  thrive 
in  the  face  of  the  stressor  (Carver,  1998;  Mancini  &  Bonanno,  2010). 
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Figure  3.  Model  for  Training  Resilient  Decision  Making 

In  order  to  support  resilience  training,  it  was  proposed  that  adaptability  and  bounce-back  are  components 
of  a  performer’s  resilience.  Specifically,  resilience  can  be  modeled  as  a  function  of  1)  the  time  needed  to 
recognize  that  the  cues  have  changed  and  the  need  to  change  action,  which  aligns  with  the  initial 
performance  drop  resulting  from  a  stressor  -  the  adaptability  phase  and  2)  the  time  to  consider  decision 
alternatives  and  act  appropriately  to  the  cue  -  the  bounce-back  phase.  A  person’s  level  of  adaptability  is 
represented  by  the  initial  performance  drop  resulting  from  the  stressor,  wherein  a  highly  adaptable  person 
is  able  to  recognize  the  stressor  and  need  to  change  course  of  action  (see  adaptability  phase  in  Figure  3). 
This  includes  accurately  assessing  best  case/worst  case/most  likely  case  as  a  result  of  exposure  to  the 
stressor,  identifying  performance  elements  required  to  achieve  successful  performance  and  recognizing 
when  adjustment  from  one  strategy  to  another  is  needed  to  address  the  condition  represented  by  the 
stressor  (Grisogono,  2006).  Adaptability  in  the  early  stages  of  the  DM  process  (Observe/Orient),  wherein 
an  individual  is  perceiving  environmental  cues  and  assessing  if  what  they  represent  is  critical.  Once  a 
person  has  recognized  the  effects  of  a  cue  and  identified  strategies  to  address  what  it  represents,  the 
second  phase  of  resilience  kicks  in  -  a  person’s  ability  to  recover  from  the  stressor  -  a.k.a.  the  bounce- 
back  phase  (see  bounce-back  phase  in  Figure  3).  A  person’s  ability  to  bounce-back  is  represented  by 
both  the  amount  and  rate  of  bounce-back,  wherein  a  highly  resilient  person  is  able  to  1)  maintain  or 
regain  balance  physiologically  in  the  presence  of  the  stressor,  2)  maintain  or  regain  focus  and  change 
their  thoughts/behaviors  in  a  positive  manner  as  needed  to  complete  tasks  despite  presence  of  stress, 
and  3)  recover  from  or  adjust  to  the  stressor,  thereby  maintaining  or  regaining  effective  performance 
levels  across  a  range  of  tasks,  situations,  and  conditions  (Lazarus,  1966).  Bounce-back  is  therefore 
critical  for  the  latter  stages  of  the  DM  process  (Decide/Act)  wherein  an  individual  is  now  evaluating 
decision  alternatives  under  the  conditions  that  the  stressor  represents  and  selecting  and  executing 
courses  of  actions  predicted  to  be  most  successful  in  the  face  of  the  stressor. 

The  underlying  assumption  of  this  model  is  that  although  many  competent  decision  makers  have  been 
trained  to  adequate  levels  of  DM  performance  void  of  stressors,  when  they  encounter  a  stressor,  their 
performance  may  be  impacted  in  a  variety  of  ways  depending  on  their  resilience  -  i.e.,  their  adaptability 
and  ability  to  bounce-back  from  stress.  It  is  theorized  that  monitoring  an  individual’s  DM  performance, 
both  process  (e.g.,  DM  stages  such  as  OODA)  and  outcome  level,  in  hand  with  their  physiological  stress 
responses  (e.g.,  heart  rate,  galvanic  skin  response)  will  enable  progress  along  the  theorized  curve  to  be 
assessed  and  their  resilience  pathway  (i.e.,  succumb,  survive,  etc.)  to  be  diagnosed.  Diagnosis  of  this 
pathway  and  where  breakdowns  in  the  DM  process  occur  will  facilitate  delivery  of  individualized  learning 
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strategies  targeted  to  build  resilience  (i.e.,  adaptability  and  bounce-back)  into  the  DM  process  and  trainee 
in  general. 

One  of  the  two  objectives  of  resilience  training  should  be  enhancing  Marine’s  adaptability.  Specifically, 
given  the  complex  and  continually  changing  environments  that  Marines  are  required  to  operate  in,  a 
critical  component  to  timely  and  accurate  decision-making  is  the  rate  at  which  they  are  able  to 
operationally  recognize  the  need  to  change  a  course  of  action  (COA)  based  on  the  underlying  causes  of 
the  experienced  cues.  The  goal  of  adaptability  training  should  be  to  train,  and  in  turn  measure,  a 
Marine's  ability  to  engage  ‘intelligent’  context-appropriate  and  flexible  assessment  of  the  environment,  be 
robust  to  adverse  and  stressful  events  that  represent  danger,  create  -  when  necessary  -  new  strategies 
in  real-time,  as  well  as  to  learn  from  and  adapt  to  lessons-learned  via  experience  (i.e.,  encode  information 
about  the  past  and  use  it  to  be  more  effective  in  the  face  of  future  stressors).  The  other  objective  of 
resilience  training  should  be  to  enhance  the  ability  of  a  Marine  to  bounce-back  from  the  impact  of  stress 
by  developing  physiological  and  psychological  coping  strategies  that  support  recovery  and  effective 
performance  despite  the  presence  of  stressors.  This  should,  in  turn,  allow  for  selection  from  among 
plausible  COAs  while  preferentially  retaining/discarding  variations  that  enhance/decrease  probability  of 
success  (Grisogono,  2006).  Over  time  the  performer  should  internalize  variations  that  tend  to  increase 
the  probability  of  success,  thus  becoming  a  resilient  decision  maker. 

Figure  4  summarizes  learning  strategies  which  can  be  leveraged  to  enhance  adaptability  and  bounce- 
back.  Once  effectively  integrated  into  a  training  regime,  such  training  should  provide  individuals  with  the 
ability  to  respond  quickly  and  intelligently  to  constantly  changing  and  stressful  environments  by  thinking 
critically  and  flexibly,  being  comfortable  with  ambiguity  and  decentralization  of  control,  dealing  with 
uncertainty  and  risk,  and  rapidly  recovering  and  adjusting  based  on  a  continuous  assessment  of  the 
situation  (Wong,  2004).  All  of  these  skills  are  critical  in  today’s  military. 


22 


□ESIGNinteractive 


Design  Interactive,  Inc. 


Contract#:  N00014-12-G-0427 


re 

> 

0) 

-I 

0) 

o 

c 

re 


o 

r 

re 

a. 


High 

Adaptability 


Thrive 


Expert 


Recover 


Survive 


Novice 


Succumb 


fsilience  =  Adaptability  ♦  Bouncfrbjick 


f  Observe 

f  Orient  A 

Detect 


Compr 


Reca 


Perce 


Feature 

Matchlm 


CjCCt 


Evaluate  I 


Adaptability  Training  Strategies 
Metacogmtion  Strategies 
Self-awareness 
Self  regulation 
Mindfulness  Training 
Cognitive  Appraisal 
Increase  Predictability  of  Stimuli 

Incorporate  Real  World  Cues 
Increase  Experience 


BouncebacK  Training  Strategies 
Stress  Inoculation  Training 

Incorporate  Stress  Into  Training 
Build  mental  Toughness-  Identify 
Negative  Reactions 

Biofeedback 


Figure  4.  Integrated  Model  of  Resilience,  Decision  Making,  and  Learning  Strategies 


3.3  Empirical  Research 


3.3.1  Experiment  1 


The  first  experiment  under  the  STAR-DM  effort  was  designed  to  address  the  following  research 
questions: 

1 .  Can  stress  be  induced  with  VBS2  scenarios,  and  if  so,  how  does  it  compare  to  the  gold  standard 
socio-evaluative  stressor? 

2.  Which  physiological  measures  are  most  indicative  of  stress  and  resilience? 
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3.  Which  DM  measures  are  most  discriminative/predictive? 

The  gold  standard  socio-evaluative  stressor  under  study  was  the  Trier  Social  Stress  Test  (TSST).  Based 
on  the  foundational  research,  it  was  hypothesized  that  the  TSST  would  induce  the  highest  amount  of 
stress  in  participants  and  that,  given  the  additive  nature  of  stress  events,  the  inclusion  of  a  socio- 
evaluative  stressor  prior  to  STAR-DM  scenarios  would  lead  to  higher  levels  of  stress  experienced  during 
the  training  simulations. 

3.3. 1.1  Methods 

Forty  people,  33  males  and  7  females,  ages  18  to  35  participated  in  the  study.  Participants  were  split  into 
two  groups,  a  control  group  who  only  received  the  VBS2-based  complex,  military-relevant  simulation 
scenarios  and  the  experimental  group  who  received  the  TSST  prior  to  the  scenarios.  The  VBS2  scenarios 
were  designed  to  be  increasing  in  difficulty  and  level  of  stress.  In  order  to  assess  the  stress  response  and 
resilience  of  participants,  performance  measures  (statistics  from  the  VBS  scenarios  on  how  well  mission 
goals  were  achieved),  physiological  measures  (e.g.,  heart  rate  variability),  and  subjective  stress 
measures  (State  Trait  Anxiety  Inventory;  STAI)  were  collected.  An  overview  of  the  experimental 
procedure  and  timeline  is  outlined  in  Figure  5  below.  For  additional  details  on  the  experimental  procedure, 
methods,  and  scenarios  see  Winslow  et  al.,  2015. 
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Figure  5.  Experimental  Procedure  and  Timeline  per  Group 


3.3. 1.2  Results 

A  summary  of  the  key  results  from  the  experiment  follows.  Further  discussion  of  the  statistical  analyses 
performed  and  results  can  be  found  in  Winslow  et  al.,  2015. 
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Figure  6.  Experiment  1  STAI  Results  per  Group 


Figure  6  shows  the  average  value  of  perceived  stress  for  each  group  at  each  stage  of  the  experiment 
[after  the  socio-evaluative  stressor  (which  for  the  control  group  was  a  placebo  version  of  the  TSST)  and 
each  VBS2  scenario]  where  the  dashed  line  indicates  the  average  overall  baseline  value.  As  predicted, 
the  TSST  induced  the  highest  level  of  perceived  stress.  The  VBS2  scenarios  induced  low  to  moderate 
levels  of  perceived  stress,  with  those  who  received  the  TSST  experiencing  higher  levels  of  perceived 
stress  during  the  subsequent  scenarios.  Between  groups,  the  experimental  group  reported  significantly 
higher  self-reported  anxiety/stress  after  the  TSST  (p  <  0.001)  and  scenarios  1  (Person  of  Interest;  POI),  2 
(Sleight  of  Hand;  SOH),  and  4  (Helicopter  Down;  HeloDown)  (all  with  p  <  0.05)  than  the  control  group. 
Within  groups,  the  control  group’s  perceived  stress  during  the  scenarios  was  not  statistically  significant 
from  baseline  with  the  exception  of  the  third  scenario  (Clandestine  Demolition;  ClanDemo).  For  the 
experimental  group,  a  statistically  significant  increase  from  baseline  was  observed  during  the  TSST, 
scenario  3  (ClanDemo)  and  scenario  4  (HeloDown)  (both  p<0.05). 
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10 


Figure  7.  Experiment  1  Performance  Results  per  Group 


As  predicted,  performance  decreased  (see  Figure  7)  across  VBS2  scenarios  from  high  to  low  levels  of 
difficulty/assumed  stress  with  the  exception  of  one  scenario  (scenario  5  -  Assassin).  There  was  not  a 
statistically  significant  difference  observed  between  the  groups  in  terms  of  performance  measures. 
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Figure  8  Experiment  1  EDR  Physiological  Results  per  Group 

Figure  8  displays  the  average  electrodermal  response  per  group  at  each  point  during  the  experiment  with 
the  dotted  line  representing  the  average  baseline  value.  As  predicted,  the  TSST  induced  the  highest 
levels  of  physiological  response  and  the  VBS2  scenarios  tend  to  increase  from  low  to  mod/high  levels. 
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The  experimental  group  experienced  significantly  higher  physiological  response  than  the  control  group  (p 
s  0.05)  during  the  TSST  and  scenarios  3  and  4. 


Figure  9.  Experiment  1  Average  Normalized  Stress  Score  by  Group  During  Baseline,  TSST,  and  VBS2 

Scenarios 

A  physiological  stress  score  was  defined  using  a  normalized  combination  of  heart  rate  and  skin 
conductance  level  (see  Dechmerowski  et  al,  2014  for  more  information)  and  the  pattern  of  those  scores 
was  analyzed  over  the  course  of  the  experiment.  Subjects  were  grouped  into  the  resistant,  resilient,  and 
recovery  trends  as  defined  by  Norris  et  al.,  2009  (see  Figure  9  above).  The  data  was  analyzed  to  see  if  it 
is  possible  to  predict  an  individual’s  resilience  classification  and  see  how  that  classification  affected  their 
performance.  Though  inconclusive  and  requiring  further  study,  results  indicated  that  resilience  can 
potentially  be  predicted  using  baseline  cortisol  and  STAI  data  (Fig  10).  Results  also  indicate  that  an 
individual’s  resilience  level  is  potentially  predictive  of  performance. 
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Figure  10:  Decision  Boundaries  of  Stochastic  Gradient  Descent  Linear  Classifier  of  Resilience  Group 

Using  Baseline  Cortisol  and  Baseline  STAI. 
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In  addition  to  studying  the  physiological  measures  and  overall  mission  performance,  an  analysis  on  the 
OODA  process-level  DM  measures  was  conducted  to  evaluate  if  those  measures  are  good  predictors  of 
overall  outcome  performance.  These  measures  were  shown  to  account  for  a  significant  amount  of 
decision  outcome  performance  with  an  R2  values  >  0.7  in  13  of  the  25  events  and  between  0.3  and  0.39 
in  11  out  of  25  events,  with  the  Decide  and  Act  measures  being  significant  predictors  in  a  half  or  more  of 
the  events. 

3.3. 1.3  Discussion 

The  results  of  this  laboratory  study  support  the  hypothesis  that  in  order  to  induce  large  amounts  of  stress 
during  simulation  based  training,  a  socio-evaluative  stressor  followed  by  simulations  specifically  designed 
to  incorporate  stressful  events  are  needed.  The  results  show  that  simulation  based  stressors  alone  are 
likely  not  going  to  illicit  a  significant  stress  response  but  that  the  addition  of  external  stress  sources  in 
conjunction  with  the  training  can  produce  stress  effects,  both  perceived  and  physiological,  that  last  for 
extended  periods. 

As  the  difficulty  and  stress  increased  over  the  scenarios  the  overall  mission  performance  scores  tended 
to  decrease  except  on  the  final  scenario.  Though  that  scenario  was  initially  rated  high  in  terms  of  stress 
and  difficulty,  after  conducting  the  experiments  and  getting  feedback,  it  was  determined  that  the 
interaction  required  with  the  scenario  was  much  less  complex,  thus  likely  leading  to  reductions  in 
perceived  stress.  Additionally,  there  was  not  found  to  be  a  significant  difference  in  mission  performance 
between  the  control  and  experimental  groups  over  the  scenarios,  thus  indicating  that  the  increased  stress 
induced  by  the  incorporated  TSST  stressor  was  not  sufficient  enough  to  lead  to  performance  decrements 
in  those  scenarios. 

One  potential  limitation  of  the  findings  is  that  the  test  was  conducted  with  non-military  personnel  thus 
drawing  into  question  whether  the  findings  would  apply  to  Marines.  Part  of  the  impetus  of  the  subsequent 
field  study,  which  will  be  described  later  on,  was  to  evaluate  that  question, 

3.3. 1.4  Conclusions 

In  summary,  the  key  results  from  the  experiment  were: 

•  Stress  can  be  induced  in  a  simulation  training  context.  However,  the  inclusion  of  a  socio- 
evaluative  stressor  is  key.  In  order  to  incorporate  this  stressor  into  the  STAR-DM  training,  the 
training  process  includes  a  step  where  the  Squad  Leader  has  to  plan  the  mission  and  brief  the 
room  and  then  receive  feedback  from  a  superior.  Additionally,  they  receive  an  AAR  debrief  of 
their  performance  in  front  of  their  squad. 

•  Stress  can  be  measured  with  the  use  of  physiological  sensors  and  the  STAI.  In  terms  of 
physiological  measures,  electrodermal  response  and  heart  rate  are  the  best  predictors.  Those 
measures  can  be  combined  into  an  operational  stress  index  which  can  be  used  to  evaluate  an 
individual’s  stress  level  throughout  training. 

•  Resilience  can  be  quantified. 

•  There  is  the  potential  to  predict  resilience  based  on  baseline  values  of  cortisol  and  STAI. 

•  An  individual’s  resilience  level  is  potentially  predictive  of  their  performance. 

•  Overall  decision-making  performance  at  the  process  level  can  be  measured. 

3.3.2  Experiment  2 

The  primary  objective  of  the  second  laboratory  experiment  was  to  evaluate  the  effectiveness  of 
incorporating  training  interventions  into  the  STAR-DM  training  framework.  The  research  question  was: 

•  Which  training  intervention,  between  learning  strategies  that  target  DM  skills  or  those  that  target 
stress  skills,  is  most  effective  at  improving  DM  under  stress? 

3.3.2. 1  Methods 

The  study  consisted  of  63  participants,  all  male,  between  the  ages  of  18  and  30.  One  important 
demographic  of  the  participant  pool  was  computer  gaming  experience  which  was  quite  varied  across  the 
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group:  9  participants  had  zero  or  low  experience,  14  had  moderately  low  experience,  10  had  moderately 
high  experience,  and  9  had  high  experience. 

Experimental  Design  and  Measures 

The  experimental  design  is  shown  in  Figure  11.  There  were  three  study  conditions;  a  control  group,  a 
group  who  received  DM  learning  strategies,  and  a  group  that  received  stress  learning  strategies  with 
biofeedback  (BF).  All  participants  completed  the  same  series  of  steps  throughout  the  study:  Introduction, 
VBS2  familiarization  training,  pretest  (socio-evaluative  stressor  and  VBS2  scenario),  education  phase, 
acquisition  (practice)  phase,  application  1  and  2  (each  with  a  different  VBS2  scenario  plus  feedback),  and 
posttest  (socio-evaluative  stressor  and  VBS2  Scenario).  Two  different  socio-evaluative  stress  induction 
techniques  were  used;  the  TSST  and  the  Socially-Evaluated  Cold  Pressor  Test  (SECPT).  Both  the  stress 
induction  techniques,  and  the  first  and  last  VBS2  scenarios  used  in  the  pre  and  posttest,  were 
counterbalanced  to  account  for  potential  order  effects.  The  red  “C”  in  the  figure  denotes  the  points  during 
the  experiment  that  cortisol  samples  were  taken.  Table  6  provides  a  description  of  the  three  study 
conditions.  For  more  details  on  the  experimental  design  and  test  procedure  please  see  Carroll  et  al.  2015. 
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Figure  1 1.  Experiment  2  High  Level  Experimental  Procedure 


Table  6.  Description  of  Experimental  Conditions 


Cond. 

Education 

(15  min  ppt.) 

Acquisition 
(10  min  Practice) 

Application 

(1  hr,  2  VBS2  Scens) 

Control 

Presented  Simple  Triage  and  Rapid 
Treatment  (START)  Combat  Triage 
decision  making  strategy,  including 
steps  for  how  to  determine  priority  of 
treatment  based  on  health  condition 

Practiced  making  triage 
decisions  using  the 
START  strategy 

Received  outcome 
feedback  after  each 
scenario  based  on 
observer  checklist. 

Decision 

Making 

Process 

Feedback 

Strategy 

Presented  concepts  of  process  level 
feedback,  the  OODA  loop,  the  impact 
of  stress  on  each  stage  of  OODA, 
performance  strategies  for  improving 
the  OODA  loop  and  error 
management  instructions 

Practiced  interpreting 
and  delivering  process 
level  feedback  by 
analyzing  previous 

VBS2  situations 

Received  outcome  and 
process  level  feedback 
after  each  scenario 
based  on  observer 
event-based  checklist 

Stress 

Bio- 

Feedback 

Strategy 

Presented  concepts  of  stress,  how  it 
affects  the  body  physiologically,  and 
biofeedback  (how  it  works  and 
examples).  Tactical  breathing 
technique  presented  via  mobile 
application  -  sound  of  an  ocean  wave 
presented  to  pace  breathing. 

Practiced  tactical 
breathing  while 
recalling  event  in  life 
that  makes  them 
anxious.  Biofeedback 
provided:  sound  of 
ocean  wave  when 
physiological  response 
above  baseline 

Received  outcome 
feedback  after  scenarios 
based  on  observer 
checklist  and 
biofeedback  during 
stressor/scenario:  sound 
of  ocean  wave  when 
physio  response  above 
baseline  (tactical 
breathing  reminder) 
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The  primary  experimental  measures  were: 

Physiological  &  Psychological  Stress  Response  Measures: 

•  Biopac  MP-150: 

•  Electrodermal  Activity  (EDA)  -  Non  dominant  hand 

•  Electrocardiogram  (ECG)  -  3  Lead  (Heart  Rate  (HR)  /  HR  Variability) 

•  Electromyography  (EMG)  -  T rapezius 

•  Respiration  Rate  (Chest  Strap) 

•  Physiological  Stress  Level:  Aggregate  of  electro-dermal  and  cardiovascular  metrics 

•  Saliva:  Cortisol  (4  times  during  study) 

•  State  Trait  Anxiety  Inventory  (STAI)  -  State  portion 

Decision  Making  (DM)  Performance:  Effectiveness  of  DM  process  calculated  by  averaging  below  scores 
across  DM  process  for  each  event  (Event  Level  DM  score)  and  across  a  scenario  (Scenario  DM  Score): 

•  Observe:  Did  participant  observe  decision  critical  cues? 

-  Score  of  critical  areas  of  interest  that  they  focused  on  (observer  checklist) 

•  Orient:  Did  participant  assess  situation  effectively? 

-  Score  of  critical  situation  factors  reported  (observer  checklist) 

•  Decide:  Did  participant  consider/select  effective  course(s)  of  action? 

Effectiveness  score  of  COA  reported  (observer  checklist) 

•  Act:  Did  participant  effectively  execute  course  of  action 

Effectiveness  score  of  COA  executed  (observer/system) 

Mission  Performance:  Calculated  by  aggregating  the  below  scores  using  the  following  method:  1)  scores 
normalized  to  data  point  between  0-10  based  on  sample  data  range,  2)  scores  weighted  based  on 
importance  in  each  scenario,  3)  weighted  scores  combined: 

•  Mission  Success  (if  mission  objectives  were  met  per  the  observer  checklist) 

•  Time  to  complete  mission  (minutes,  system  collected) 

•  Survivability  (number  of  shots  taken;  system  collected) 

•  Civilians  killed  (number,  system  collected) 

Since  many  of  the  DM  measures  relied  on  data  collected  using  the  observer  checklist  (see  example  in 
Figure  12  below),  an  interrater  reliability  (IRR)  analysis  was  performed  to  ensure  consistency  in  those 
collected  measures.  The  process  for  collecting  data  using  the  checklist  was  that  one  of  three  observers 
would  watch  a  participant  navigate  through  the  scenario  and  for  each  event  would  check  off  the 
appropriate  boxes  corresponding  to  their  statements  and  actions.  Note  that  participants  were  asked  to 
talk  thorough  their  thoughts  out  loud  so  that  the  observers  could  determine  their  COA  considerations. 
Each  event  and  scenario  was  different  (e.g.  there  may  be  multiple  key  items  of  interest  to  detect  in  one 
given  event,  but  only  one  in  another),  but  all  were  scored  similarly.  Fleiss’  Kappa  (KF)  was  calculated  for 
each  of  the  four  scenarios  used  in  the  experiment  with  resulting  values  between  0.80  and  0.92.  All  of  the 
results  fall  within  either  the  generally  accepted  “strong”  or  “near  complete”  agreement  ranges,  indicating 
strong  consistency  among  the  raters. 
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Figure  12.  Example  of  Observer  Checklist 


3.3.2. 2  Results 

Preliminary  correlational  analysis  revealed  that  DM  performance  in  the  pretest  scenario  was  significantly 
correlated  with  computer  gaming  experience  (r  =  .39,  p  =.01).  This  finding  is  not  surprising  in  that  it 
makes  sense  that  those  more  comfortable  playing  computer  games  would  have  less  difficulty  interacting 
with  and  be  more  comfortable  with  the  early  scenarios  which  could  lead  to  better  performance.  Therefore, 
computer  gaming  experience  was  used  as  a  covariate  throughout  the  analysis.  Additionally,  the 
preliminary  analysis  showed  that  the  change  in  DM  performance  from  pretest  to  posttest  was  significantly 
correlated  with  the  pretest  scenario  (r  =  .47,  p  =.01).  In  other  words,  despite  efforts  to  equalize  difficulty 
and  stress  of  the  two  scenarios  (Clandestine  Demolition  (CD)  and  Helo  Down  (HD)),  one  scenario  (HD) 
had  much  lower  performance  than  the  other.  Figure  13  below  shows  the  DM  performance  of  the  two 
scenarios,  pretest  and  posttest.  Those  who  received  the  hard  scenario  (HD  shown  in  blue)  first  showed 
improvement  in  the  posttest  that  was  partially  due  to  their  posttest  scenario  (CD  shown  in  red)  being 
easier. 
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made  up  of  both  change  in 
difficulty  and  increases  in 
performance  of  some 
groups 
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Figure  13.  Decision  Making  Performance  by  Pretest  and  Posttest  Scenario 
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In  breaking  down  performance  in  terms  of  each  stage  of  the  OODA  loop,  the  differences  in  DM 
performance  can  be  better  determined.  Figure  14  shows  the  overall  average  score  per  OODA  stage  for 
the  CD  and  HD  scenarios.  The  differences  between  CD  and  HD  are  negligible  in  the  Observe  and  Orient 
stages,  but  a  big  performance  difference  is  seen  in  the  Decide  and  Act  stages  indicating  HD  is  more 
difficult.  Given  the  significance  of  pretest  scenario  in  the  correlational  analysis,  pretest  was  also  used  as  a 
covariate. 


DM  Performance  per  OODA  Loop  stage 
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Figure  14:  Decision  Making  Performance  at  Each  OODA  Loop  Stage 


Next,  the  data  was  analyzed  to  determine  the  findings  for  the  three  hypotheses  underlying  the 
experiment.  The  first  hypothesis  was  that  DM  learning  strategies  would  result  in  significant  DM 
performance  improvements  from  pretest  to  posttest  compared  to  control.  The  average  change  score  for 
each  condition  was  analyzed  (see  Figure  15)  in  order  to  evaluate  the  first  hypothesis.  When  all  conditions 
were  considered  together,  the  trial  by  condition  interaction  was  not  significant  (F(2,  58)  =  3.23,  p  =  .14,  r\2 
=  .06).  However,  when  comparing  only  the  DM  and  Control  groups  (Figure  16),  there  was  a  significant 
trial  by  condition  interaction  wherein  the  DM  group  had  significantly  greater  decision-making  performance 
improvements  than  the  control  group  (F  (1 ,  38)  =  4.94,  p  =  .03,  n2  =  -1 2). 
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Figure  15:  Average  DM  Change  Score  from  Pre  to  Posttest 
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Figure  1 6:  Decision  Making  Performance  for  Control  and  DM  Conditions 
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The  second  hypothesis  was  that  stress  learning  strategies  would  result  in  significant  reductions  in  stress 
response  (physiological  and  psychological)  from  pretest  to  posttest  compared  to  control.  As  in  Experiment 
1,  physiological  stress  was  measured  using  a  stress  score  which  is  a  classifier  developed  and  validated 
under  another  effort  (Dechmerowski  et  al.  (2014))  that  is  made  up  of  heart  rate  variability  and 
electrodermal  activity  metrics  and  is  baseline  normalized  by  individual.  Figure  17  shows  the  average 
stress  score  values  by  group  during  the  different  stages  in  the  experiment.  Though  there  was  not 
sufficient  sample  size  to  show  overall  significance  across  all  conditions,  results  suggest  both  BF  and  DM 
learning  strategies  led  to  reductions  in  stress  response  compared  to  control.  When  evaluating  the 
difference  from  pre  to  posttest  stress  response  (Figure  18),  the  BF  group  saw  a  significant  decrease  in 
physiological  stress  response  from  Pretest  Stressor  to  Posttest  Stressor  (p  =  0.024).  The  Control  group  (p 
=  0.386)  and  DM  group  (p  =  0.320)  did  not. 


Figure  1 7:  Average  Stress  Response  by  Group  by  Experimental  Stage 
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Figure  18:  Average  Stress  Response  by  Group  by  Experimental  Stage 


Cortisol  levels  compared  from  pre  to  posttest  show  that  both  the  DM  and  BF  groups  had  a  significant 
decrease  in  cortisol  response  from  pretest  to  posttest  (BF  p  =  0.047;  DM  p  =  0.010)  while  the  control 
group  did  not  (p  =  0.636).  See  Figure  19. 
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Figure  19:  Average  Cortisol  Level  by  Group  from  Pretest  to  Posttest 

When  evaluating  perceived  stress  using  the  STAI  there  was  not  a  significant  decrease  in  perceived  stress 
from  pretest  to  posttest  for  any  group  (see  Figure  20).  This  is  likely  due  to  administration  of  the  STAI  after 
the  scenario  which  in  Experiment  1  proved  to  be  a  recovery  period. 
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Figure  20.  STAI  by  Group 

The  third  hypothesis  was  that  stress  learning  strategies  would  lead  to  improved  resilience  classification 
(e.g.,  from  recovery  to  resilient)  from  pretest  to  posttest  compared  to  control.  However,  results  show  that 
there  was  not  a  significant  change  in  resilience  classification  from  pretest  to  posttest  in  any  group.  This 
indicates  that  changes  of  resilience  level  may  not  be  trainable  in  a  short-term  experiment. 


One  additional  analysis  was  performed  to  evaluate  if  there  was  an  overall  mission  performance 
improvement  in  the  BF  and  DM  groups  compared  to  control.  Although  there  was  not  enough  evidence  to 
show  statistical  significance,  participants  who  received  control  learning  strategies  had  negligible 
performance  changes,  while  there  appeared  to  be  increases  in  performance  from  participants  in  both  the 
DM  and  BF  conditions  (Figure  21). 
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Figure  21.  Mission  Performance  Change  Score  from  Pre  to  Posttest 
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3.3. 2.3  Discussion 

The  results  of  the  second  laboratory  study  support  the  hypothesis  that  the  DM  learning  strategy  which 
combined  both  process  level  and  outcome  feedback,  led  to  significantly  greater  increases  in  DM 
performance  than  the  outcome  feedback  alone  received  by  the  control  group.  This  is  in  line  with  findings 
from  the  literature  (Earley  et  al.,  1990;  Carroll,  2010)  and  provides  support  for  utilizing  such  a  strategy  to 
target  DM  performance  in  training.  The  process  level  feedback  allows  for  the  pinpointing  of  where  in  the 
DM  process  the  breakdown  occurred  so  the  performer  can  focus  their  attention  on  task  strategies  used  in 
deficient  sub  steps,  allowing  a  more  effective  and  efficient  learning  experience. 

The  inclusion  of  stress  learning  strategies  combined  with  biofeedback  reduced  the  stress  response  of 
individuals.  By  helping  the  trainee  recognize  when  they  are  experiencing  stress  and  giving  them 
strategies  to  reduce  their  stress  response,  individuals  are  able  to  better  regulate  their  stress  levels.  Both 
learning  strategies  appear  to  improve  overall  performance  but  more  research  is  needed  in  order  to  prove 
that  assertion. 

These  results  must  be  viewed  with  caution  as  there  were  several  limitations  of  the  studies.  One  limitation 
of  both  laboratory  studies  was  the  amount  of  training  time.  Given  the  limited  amount  of  time  participants 
were  available,  both  training  times  were  severely  limited.  Decision  making  is  a  complex  skill  which 
depends  heavily  on  building  up  one’s  experience  base,  something  not  possible  in  one  to  two  training 
scenarios. 

3.3.2.4  Conclusion 

The  results  from  the  second  lab  experiment  indicate: 

•  Process  level  feedback  improved  decision-making  performance, 

•  Biofeedback  reduced  stress  response, 

•  The  resilience  classification  hypothesis  was  unsupported  indicating  changes  in  resilience 
classification  level  through  training  may  require  more  time, 

•  Both  process-level  feedback  and  biofeedback  showed  promise  for  increasing  overall  mission 
performance. 

3.3.3  Field  Study 

The  STAR-DM  Field  Study  had  three  primary  research  questions  that  it  aimed  to  address: 

•  Can  stress  be  induced  in  Marines  utilizing  simulation-based  training? 

•  Does  the  STAR-DM  training  approach  improve  decision-making  performance? 

•  Do  the  performance  improvements  transfer  to  the  field? 

A  summary  of  the  experimental  methods,  design,  execution,  results  and  analysis  follows.  For  detailed 
information  see  Carroll  et  al.,  2014  &  Carroll  et  al.,  2015. 

3.3. 3.1  Methods 

Thirty  male  Marine  Corps  Sergeants  (E-5)  with  Squad  Leader  experience  enrolled  in  the  ISULC  course  at 
the  SOI-E  served  as  study  participants.  After  a  study  introduction,  all  participants  received  a  computer 
based  Situational  Judgment  Test  (SJT)  as  a  pre-test.  The  group  was  then  split  into  the  control  and 
experimental  groups  and  received  approximately  8  hours  of  training  with  the  control  group  (made  up  of  8 
leaders)  receiving  three  scenarios  created  by  the  simulation  lab  at  SOI-E  and  the  experimental  group  (5 
leaders)  receiving  five  scenarios  of  the  STAR-DM  SLTPs.  All  trainees  then  completed  a  post-test  SJT  and 
then  a  series  of  field  exercises  (FEX).  Figure  22  shows  the  experimental  flow  and  Table  7  shows  how  the 
participants  were  split  into  experimental  group  Squad  Leaders  and  teammates  and  control  group  leaders 
and  teammates. 
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Figure  22:  Field  Study  High  Level  Experimental  Procedure 


Table  7.  Group  Descriptions 


Condition 

Focus  of  Study 

Not  Focus  of  Study 

Experimental 

Experimental  Group  Squad  Leaders  (5) 

Experimental  Group  Teammates  (1 0) 

•  Acted  as  Squad  Leader  during 

•  Did  NOT  act  as  Squad  Leader  during 

training 

training  (therefore,  no  feedback 

•  Received  process  level  feedback 

received) 

•  Wore  physiological  sensors 

•  Did  not  wear  physiological  sensors 

Control 

Control  Group  Squad  Leaders  (8) 

Control  Group  Teammates  (7) 

•  Acted  as  Squad  Leader  during 

•  Acted  as  Squad  Leader  during  training 

training 

•  Received  traditional  feedback 

•  Received  traditional  feedback 

•  Wore  physiological  sensors 

•  Did  not  wear  physiological  sensors 

Measures 

The  primary  DM  measures  that  were  utilized  in  the  study  were  pre  and  posttest  competency  as  assessed 
by  SJT  ratings  compared  to  subject  matter  experts.  To  assess  training  transfer,  DM  expertise  level  was 
assessed  during  the  field  study  by  instructors  using  the  Behaviorally  Anchored  Rating  Scale  (BARS)  to 
evaluate  trainees  in  key  performance  areas  (KPAs). 

To  measure  psychological  stress,  the  STAI  was  used  to  measure  perceived  stress  and  the  Connor- 
Davidson  Resilience  Scale  (CD-RISC)  was  used  to  measure  the  trainee's  stress  coping  ability.  A  10-point 
Likert  scale  questionnaire  was  also  administered  to  capture  trainee  reactions  to  the  stressfulness  of  the 
training  they  received. 

Multiple  physiological  measures  were  collected  during  the  study  using  the  Biopac  MP-150.  However,  as 
found  in  prior  studies,  EDA  and  HRV  were  shown  to  be  the  best  measures  for  discriminating  a  stress 
response  so  these  two  measures  were  the  primary  ones  utilized. 

Procedure 

The  test  procedure  is  outlined  in  Table  8.  Training  for  both  groups  consisted  of  approximately  eight  hours 
of  VBS2  simulation-based  training  which  began  after  the  study  introduction  and  pretest.  The  STAR-DM 
and  Sim  Lab  training  approaches  are  outlined  in  Figure  23.  In  the  control  group,  an  instructor  would 
interact  with  the  simulation  to  elicit  decision  events  and  enact  COAs  that  were  chosen  by  the  squad.  An 
AAR  of  the  squad’s  response  to  the  event  and  planned  COA  was  given  by  the  instructor  after  each  event. 
In  the  STAR-DM  group,  each  squad  member  interacts  directly  with  the  VBS2  scenario  to  make  decisions 
and  enact  responses  to  events  that  occur.  After  each  event,  the  instructor  provides  overall  feedback  on 
mission  and  decision  event  performance.  They  also  provide  process  level  feedback  using  an  OODA 
checklist  to  help  trainees  understand  where  breakdowns  occur.  This  checklist  includes:  1)  cues 
detected/not  (Observe),  the  situational  factors  recognized/not  (Orient),  the  effectiveness  of  COAs 
chosen/not  (Decide)  and  the  effectiveness  of  COA  execution  (Act).  Following  the  training,  a  posttest  was 
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given.  Then  two  weeks  later,  the  participants  were  evaluated  during  two  field  exercises  for  transfer  of 
training. 


Table  8:  Field  Test  Procedure  by  Day 


Day(s) 

Activity 

1 

Study  introduction,  informed  consent  forms  and 
questionnaires  completed 

2 

SJT  pretest,  split  into  control  and  experimental 
conditions,  begin  VBS2  simulation-based  training 

3 

Complete  VBS2  training  (~8  total  hours  over  1 .5 
days),  SJT  posttest 

2  weeks  after  posttest 

4-7 

Series  of  two  field  exercises,  BARS  used  to 
assess  transfer  performance  on  KPAs 

STAR-DM  Approach 

1.  "Road  to  War"  &  Operational 
Update  -multimedia  video 

2.  Mission  Orders  -  Verbal 

3.  Mission  Planning  &  Issue 
Tasking-  work  individually, 
briefed  to  entire  room,  public 
evaluation  and  feedback 

4.  Mission  Execution  in  VBS2  - 

first  person,  dynamic, 
interactive,  stressors 
incorporated 

5.  Assessment  and  Debrief  - 

Event-based  checklist  to  assess 
and  structure  debrief 


Sim  Lab  Approach 

1.  "Road  to  War"  &  Operational 
Update  -  verbal  and  sand  table 

2.  Mission  Orders  -  Verbal 

3.  Mission  Planning  &  Issue 
Tasking-  worked  as  team, 
briefed  to  COC  over  radio, 
feedback  in  front  of  team 

4.  Mission  Execution  in  VBS2  - 

top  down  view,  static,  not 
interactive,  stress  not  the 
focus 

5.  Assessment  and  Debrief  - 

Overall  assessment  form  to 
assess  and  structure  debrief 


Figure  23:  Description  of  Training  Content  and  Structure  Per  Condition 


3.3. 3.2  Results 

Change  in  DM  competency  was  calculated  by  computing  the  difference  between  the  SJT  pre  and  posttest 
scores.  Results  indicated  that  the  groups  were  significant  (F  (3,  26)  =  3.15,  p  =  .04,  q2  =  .27),  with  the 
experimental  group  Squad  Leaders  having  the  greatest  improvements  from  pre  to  post  test.  Pairwise 
comparisons  showed  the  experimental  Squad  Leader  group  was  significantly  different  than  both  the 
experimental  teammate  and  control  teammate  groups  (both  p  <  0.05).  Control  group  Squad  Leaders  were 
also  significantly  different  than  control  group  teammates  (p  =  0.5).  Although  it  appears  that  the  STAR-DM 
leaders  had  the  greatest  improvement  in  DM  competency,  there  was  not  sufficient  sample  size  to  show 
statistically  significant  difference  from  the  control  group  leaders.  See  Figure  24  for  graph  of  results. 
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Figure  24.  Change  in  Decision  Making  Competency  as  Indicated  by  SJT  Scores. 


To  determine  if  the  SLTP  training  could  induce  stress  in  Marines,  the  mean  rate  of  EDR  for  the 
experimental  group  Squad  Leaders  throughout  the  five  segments  of  the  training  were  evaluated.  STAR- 
DM  leaders  showed  significant  increases  in  physiological  stress  response  throughout  training,  with  the 
socio-evaluative  stress  having  the  highest  impact.  As  shown  in  Figure  25,  EDR  was  significantly  higher 
than  baseline  values  (shown  as  dotted  line)  throughout  all  segments  of  the  training  (*p  <  0.05) 
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Figure  25.  EDR  Average  Values  for  Experimental  Leaders  at  Different  Points  in  the  Training. 
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HRV  was  also  analyzed  over  the  course  of  the  training.  Although  the  temporal  domain  HRV  trended 
lower  during  the  brief,  fire,  and  debrief  segments  as  compared  to  baseline,  the  differences  were  not 
statistically  significant. 

STAI  results  were  also  analyzed  for  both  experimental  leaders  (n  =  5)  and  control  leaders  (n  =  8)  to 
evaluate  perceived  stress.  Results  did  not  show  statistically  significant  effects  of  condition  (experiment 
vs.  control)  or  trial  (baseline  vs.  post  scenario).  However,  when  evaluating  the  interaction  of  trial  and 
condition  the  results  are  close  to  achieving  statistical  significance  (F(1,  10)  =  4.71;  p  =  .055,  q2  =  .32), 
Figure  26  shows  the  absolute  values  of  STAI  by  trial  and  condition  and  Figure  27  shows  the  %  change 
from  pre  to  post-test.  These  results  indicate  that  the  STAR-DM  leaders  experienced  an  increase  in 
perceived  stress  as  result  of  the  training  whereas  the  control  group  did  not. 
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Figure  26:  Pre  and  Post  STAI  Values  by  Group,  Figure  27:  STAI  Percent  Change 


In  evaluating  if  the  training  effect  would  transfer  to  the  field,  the  idea  was  to  compare  the  expertise  level 
displayed  during  the  field  exercises  between  the  control  and  experimental  group.  However,  due  to  field 
study  limitations,  it  turned  out  that  those  who  served  as  Squad  Leaders  during  the  field  exercises  were  all 
from  the  experimental  group,  thus  making  that  comparison  impossible.  In  lieu  of  that  comparison,  the 
analysis  compared  performance  differences  between  those  who  served  as  Squad  Leaders  during  the 
training  (and  therefore  were  the  focus  of  the  training)  and  those  who  were  teammates,  See  Figure  28  for 
results  of  tactical  thinking  and  adaptability  expertise  levels  by  group.  Given  the  small  sample  size,  no 
statistically  significant  differences  were  found  between  the  groups.  However,  the  effect  size  was 
moderate  indicating  that  a  larger  sample  size  might  lead  to  significance  (F( 2,  4)  =  .87,  p  =  .49,  q2  =  .30). 
Based  on  the  sample,  results  indicate  the  potential  for  supporting  SLTP  training  improvements  that 
transfer  to  the  field,  in  that  participants  who  served  as  leaders  during  the  SLTP  training  were  assessed  at 
one  whole  expertise  level  on  average  higher  than  those  who  were  teammates  (Competent  versus 
Advanced  Beginner). 
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3.3. 3. 3  Discussion 

Even  when  utilized  in  a  field  setting,  specifically  during  a  Marine  Corp  training  course,  results  show  similar 
promise  for  the  effectiveness  of  process  level  feedback  in  improving  DM  competency  of  Squad  Leaders. 
Squad  Leaders,  who  received  the  process-level  feedback,  showed  the  greatest  increases  in  DM 
competency,  although  not  statistically  significantly  greater  than  their  control  group  counterparts.  Also 
supporting  the  effectiveness  of  the  process  feedback-based  learning  strategy  is  the  increase  in  expertise 
level  displayed  by  the  experimental  group  Squad  Leaders  over  their  teammates.  Ideally,  the 
experimental  group  Squad  Leaders  would  have  been  compared  against  the  control  group  Squad 
Leaders,  but  as  with  many  field  studies,  training  requirements  had  to  be  prioritized  by  the  instructors  over 
the  experimental  design  of  the  study.  Therefore,  only  members  of  the  experimental  group  were  able  to  be 
assessed  in  the  transfer  exercise.  Even  so,  the  analysis  that  was  conducted  suggests  that  the  process 
level  feedback  not  only  impacted  performance  and  competency  levels  during  and  just  after  training,  but 
that  these  improvements  may  transfer  to  a  live  environment  several  days  after  the  training  occurred. 

Results  also  provide  support  for  the  ability  to  induce  and  measure  stress  in  Warfighters  during  simulation- 
based  training  using  the  STAR-DM  approach.  The  simulation-based  training  resulted  in  significant 
increases  in  physiological  stress  response  as  indicated  by  EDA  levels.  Further,  when  accounting  for 
individual  differences  in  baseline  perceived  stress,  the  simulation-based  training  scenarios  led  to  greater 
increases  in  perceived  stress  as  indicated  by  STAI  responses.  Interestingly,  the  training  segments  which 
incorporated  socio-evaluative  stress  resulted  in  the  greatest  physiological  responses.  This  is  a  promising 
result  for  the  military  as  incorporation  of  socio-evaluative  stress  can  be  accomplished  with  very  few 
resources  because  characteristics  of  its  implementation  tend  to  inherently  exist  in  many  commonly  used 
military  training  exercises  for  Squad  Leader  training  (e.g.  judgment  of  experts  while  briefing  plan,  flat 
affect  of  experts).  Further,  the  ability  to  utilize  simulation  to  induce  stress,  and  the  ability  to  capture  stress 
response  within  Warfighters,  provides  the  military  with  viable  training  tools  for  enhancing  decision-making 
under  stress.  The  capability  to  identify  times  when  a  Warfighter  is  experiencing  increased  stress  response 
allows  instructors  to  pinpoint  factors  contributing  to  performance  decrements  (e.g.,  competent  performer, 
but  performance  unravels  under  stress)  so  as  to  effectively  tailor  future  training  to  address  each  person’s 
individual  shortcomings  (e.g.,  needs  additional  practice  performing  skills  in  stressful  environment). 

Despite  significant  increases  in  perceived  and  physiological  stress  response  in  the  experimental  group, 
participants  rated  the  training  as  being  of  low  to  moderate  stressfulness.  This  reaction  was  likely  due  to 
the  standard  against  which  the  training  was  being  compared  (e.g.  live  environments).  However,  the 
physiological  data  provides  evidence  that  the  training  did  induce  a  degree  of  stress  indicating  that  self- 
report  measures  may  not  always  portray  the  complete  story;  more  objective  measures  are  needed  to  fully 
understand  what  is  actually  happening  within  the  Warfighter.  To  better  understand  the  perceived  stress  of 
the  simulation-based  training  compared  with  theoretically  less  stressful  (e.g.  classroom  lecture)  and  more 
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stressful  (e.g.  live  exercises),  future  research  should  administer  the  STAI  after  different  types  of  training  in 
the  training  continuum.  This  would  provide  more  insight  into  whether  incorporating  simulation-based 
training  into  the  middle  of  the  training  continuum  does  indeed  support  increasing  stress  levels  throughout 
the  training  continuum. 

These  results  must  be  viewed  with  caution  as  there  were  several  limitations  of  the  studies.  One  limitation 
was  the  amount  of  training  time,  which  was  limited  due  to  the  amount  of  time  available  within  the  training 
schedule.  Decision  making  is  a  complex  skill  which  depends  heavily  on  building  up  one’s  experience 
base,  something  that  is  difficult  to  accomplish  within  one  to  two  training  scenarios.  A  second  limitation 
with  the  field  study  was  the  lack  of  control  in  the  study  design  and  execution.  As  a  result,  there  were 
several  differences  between  the  experimental  and  control  group  training,  in  addition  to  whether  the 
feedback  was  process  level  or  outcome  feedback.  It  is  not  expected  that  these  differences  made  a  huge 
impact  on  the  results,  but  it  must  be  noted  that  they  exist.  Finally,  the  field  study  had  a  small  number  of 
participants,  limiting  the  power  of  the  study. 

3.3.3.4  Conclusion 

Results  suggest,  but  do  not  definitively  prove,  that  the  STAR-DM  simulation  training  packages: 

•  Can  induce  a  significant  physiological  stress  response,  even  in  experienced  Marines, 

•  Can  improve  decision-making  performance  during  training, 

•  Can  improve  decision-making  performance  in  stressful  field  exercises. 

3.3.4  TEE 

In  order  to  determine  if  the  potential  gains  demonstrated  in  the  laboratory  and  field  studies  would  transfer 
to  the  real  world  environment,  a  training  effectiveness  evaluation,  or  TEE,  was  designed.  The  original 
design  included  a  control  group  and  a  group  that  would  undergo  the  STAR-DM  SLTP  training  packages 
with  both  the  learning  strategies  and  biofeedback  tested  in  Experiment  2  incorporated  into  their  training. 
Pre  and  post-test  measures  would  be  conducted  prior  to  and  after  the  training.  Subsequent  to  receiving 
the  training,  both  groups  would  then  conduct  a  high-stress,  live  training  exercise  and  their  performance 
during  the  exercise  would  be  measured  and  compared  to  see  if  improvements  could  be  found  in  the 
experimental  group. 

However,  during  the  course  of  the  STAR-DM  program,  additional  objectives  emerged  for  the  culminating 
event.  In  addition  to  wanting  to  understand  the  effectiveness  of  the  STAR-DM  simulation  training 
packages,  it  was  also  desired  to  evaluate  the  effectiveness  of  other  training  environments.  Therefore,  the 
primary  research  question  that  emerged  was:  How  does  the  training  effectiveness  of  the  following  training 
environments  compare;  Classroom  training  (in  the  form  of  TDGs),  Simulation  training  (STAR-DM  SLTPs), 
training  on  a  basic  non-instrumented  range  (such  as  62  Area  RUF  at  Camp  Pendleton),  training  on  a 
more-sophisticated,  instrumented  range  (such  as  Kilo2  at  Camp  Pendleton),  and  training  at  the  NT. 

In  terms  of  comparing  the  five  training  environments,  a  prioritized  list  of  study  outcomes  was  developed. 
The  four  primary  outcomes  selected,  in  prioritized  order,  were  return  on  training  investment  (ROTI), 
training  transfer  impacts,  training  impacts,  and  calculation  of  the  percent  of  training  objectives  that  can  be 
trained  in  the  given  environment.  ROTI  is  an  objective  method  for  comparing  training  value  and  costs  in 
order  to  quantify  benefit  gained  through  an  investment  in  training.  As  a  primary  measure  for  the  TEE,  an 
overall  ROTI  score  for  each  environment  would  be  determined  by  aggregating  weighted  measures  of 
performance,  cost  and  schedule  impacts  to  come  up  with  an  “apples  to  apples”  comparison  across 
environments.  To  determine  the  impact  of  the  training  types  on  transfer  to  the  real  world,  the  following 
four  items  would  be  evaluated  in  a  hyper-realistic  environment:  decision-making  performance,  task 
performance,  stress  inoculation,  and  expertise  level  displayed  during  the  event.  The  impact  of  the  training 
session  would  be  measured  by  collecting  data  pre-training  and  post-training  on  the  following:  stress 
induction,  sense  of  presence,  engagement,  DM  competency,  task  performance,  and  expertise  level 
displayed. 
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The  desired  participant  group  for  the  study  was  nine  Marine  Infantry  squads.  As  the  Squad  Leaders  are 
the  focus  of  the  training  this  would  only  allow  for  two  Squad  Leaders  for  four  conditions  and  one  in  one  of 
the  conditions  which  is  not  ideal.  However,  due  to  typical  unit  sizes,  larger  numbers  of  squads  were  not 
deemed  feasible.  The  initial  design  for  the  TEE  is  outlined  in  Figure  29.  The  first  day  would  include  all 
disclosures  and  forms  per  the  study  protocol  as  well  as  collecting  baseline,  pre-training  measures  via 
surveys,  SJTs,  and  physiological  sensors.  During  the  second  day,  participating  squads  would  be  split  into 
one  of  the  five  training  groups  and  undergo  a  full  day  of  training.  The  training  in  each  group  would  be 
designed  to  be  as  similar  as  possible,  given  the  training  condition,  in  terms  of  training  objectives  and  the 
decision  areas  being  targeted.  On  the  third  day  the  post-training  measures  would  be  collected.  Then  on 
the  last  day,  participants  would  complete  one  or  two  live  exercises  in  the  transfer  environment.  The 
preliminary  plan  for  the  transfer  environment  was  to  conduct  a  live  exercise  at  Strategic  Operations 
(STOPS)  in  San  Diego,  California.  STOPS  is  a  military  training  company  that  offers  hyper-realistic  training 
complete  with  movie  set-like  effects  and  trained  role  players.  The  goal  of  utilizing  STOPS  for  the  transfer 
environment  would  be  to  provide  an  environment  as  close  to  real-world  combat  as  possible  for  evaluating 
training  effectiveness. 


Groups 


Group  1: 
IIT 


Group  2: 
Kilo2 


Group  3: 
RUF 


Group  4: 
Sim 


Group  5: 

Class 

room 


Pre  Test 

Training 

Post  Test 

Transfer 

(Day  1,  3  hrs) 

(Day  2,  full  day) 

(Day  3,  2  hrs) 

(Day  4,  full  day) 

•  3-4  Scenarios  at  IIT* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 


All  Trainees: 

•  Pre-training 
Surveys 

•  Pre-training 
measures 
(Situational 
Judgment  Test; 
SJT,  Expertise, 
Physio) 


•  3-4  Scenarios  at  Kilo2* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 

•  3-4  Scenarios  at  RUF* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 

•  3-4  STAR-DM  Scenarios 
at  IIT  or  Sim  lab* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 

•  3-4  Marine-led  Tactical 
Decision  Games* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 


All  Trainees: 

•  Post-training 

measures  (Surveys, 
SJT,  Expertise) 


All  Trainees: 

•  Each  group  will 
rotate  through 
Strategic  Operations 
and  complete  the 
same  1-2  scenarios 

•  Transfer  Measures 
(Performance, 
Survey,  Physio) 


Potential  to  Consolidate 
days  2  &  3,  resulting  in  3 
day  data  collection 


*  All  training  scenarios  will  have  similar  training  objectives  and  decision  areas  targeted. 


Figure  29:  Initial  TEE  Design 


Unfortunately,  the  program  team  and  sponsors,  were  unable  to  find  a  Marine  unit  capable  of  supporting 
the  test  effort.  Many  attempts  were  made  over  the  course  of  a  year  and  a  half  to  set  up  the  evaluation  but 
none  were  successful.  Given  that  the  significant  time  requested  for  the  study  was  one  of  the  difficulties 
encountered  in  finding  an  available  unit,  the  proposed  study  design  was  modified  and  reduced  in  scope  to 
decrease  the  time  required  (see  Figure  30).  However,  the  team  was  still  unsuccessful  in  identifying  an 
available  unit  to  support.  Since  the  culminating  evaluation  of  STAR-DM  was  unable  to  be  executed,  the 
full  training  framework  proposed  in  the  following  section  was  unable  to  be  evaluated  in  its  entirety,  though 
the  pieces  of  the  framework  were  evaluated  separately  in  the  field  and  laboratory  experiments.  Thus, 
while  there  is  evidence  to  suggest  there  are  multiple  benefits  to  the  STAR-DM  training  approach  and  the 
SLTPs,  a  complete  assessment  was  not  possible. 
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DAY  1  -  Training 

Groups 

Pre  Test 
(2  hrs.) 

Training 
(4  hrs.) 

Post  Test 
(1  hr.) 

Group  1 
(2  squads): 

No  training 

All  Trainees: 

•  See  note 

Group  2 
(2  squads): 

62  Area  RUF 

•  Pre¬ 
training 
Surveys 

•  Situational 

•  3  Scenarios  at  RUF* * 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 

All  Trainees: 

•  Post¬ 
training 

Group  3 
(3  squads): 
Simulation 

Judgment 
Test  (SJT) 
on 

computers 
•  Physio  - 
Baseline 

•  3  STAR-DM  Scenarios  at  IIT 
or  Sim  lab* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 

Surveys 
•  SJT  on 
computers 

Group  4 
(2  squads): 
Classroom 

•  3  Marine-led  Tactical 

Decision  Games* 

•  During  Training  Measures 
(Perform.,  Survey,  Physio) 

DAY  2  -  Transfer 

Transfer 
(full  day) 


All  Trainees: 

•  Each  group  to  rotate  through  I  IT 
and  complete  the  same  2-3 
scenarios 

•  Transfer  Measures 
(Performance,  Survey,  Physio) 


Note:  The  group  the  does  not  receive  training  per  the  experiment  will  conduct  their  transfer  scenarios  at  the  I  IT 
on  day  1  during  the  training  time  and  on  day  2  they  can  receive  additional  training,  as  desired,  in  the  form  of 
TDGs  and/or  STAR-DM  simulation  scenarios 


Figure  30.  Modified  TEE  Design  to  Reduce  Time  Required 


3.4  Final  Training  Framework 


Based  on  experimental  results,  a  number  of  alterations  were  made  to  the  STAR-DM  initial  conceptual 
model  (section  3.2.5).  Due  to  the  difficulty  in  quantifying  OODA-based  performance  metrics,  including 
real-time  metrics,  and  the  absence  of  a  correlation  between  stress  and  performance  as  defined  in  VBS2 
scenarios,  a  stress-based  temporal  model  of  performance  was  chosen,  with  real-time  stress  metrics 
associated  with  cardiovascular  and  electrodermal  activity,  which  was  validated  in  Experiment  1  and  a 
separate  Veterans  Affairs  (VA)  randomized  controlled  trial  (Winslow,  2016)  as  the  basis  for  determining 
the  resilience  trajectories  individuals  undergo  during  stress  (see  Figure  31,  Norris  Soc  Sci  Med  2009). 
The  following  alterations  were  made  to  the  Norris  (2009)  model  shown  in  Figure  31  to  form  the  basis  of 
the  STAR-DM  stress  conceptual  model: 

•  Temporal  metrics  are  in  the  minutes  to  hours  timeframe,  rather  than  the  months  to  years  frame, 

•  Relapsing-remitting  and  delayed  dysfunction  trajectories  are  not  included;  they  were  not  observed 
in  the  STAR-DM  studies  or  the  studies  performed  by  Norris,  et  al., 

•  Real-time  physiological  stress  metrics,  rather  than  PTSD  scores  are  utilized  as  the  dependent 
variable. 
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Figure  31:  Trajectories  Following  Exposure  to  Severe  Stress.  Norris,  Soc  Sci  Med  2009. 

Our  data  suggests  that  resilience  is  best  understood  and  measured  as  a  trajectory,  indicative  of  increased 
physiological  (cardiovascular,  electrodermal)  activity  associated  with  severe  stress,  followed  by  a  return  to 
baseline  within  tens  of  minutes.  Recovery  is  indicative  of  increased  physiological  activity  with  stress, 
followed  by  a  reduction  after  the  stressor  without  returning  to  baseline,  whereas  chronic  dysfunction  is  not 
associated  with  decreased  physiological  reactivity  following  stressor  removal.  Resistance  is  defined  by  no 
measurable  increase  in  physiological  reactivity  to  stress.  All  are  dependent  on  an  individualized  baseline 
physiological  signature. 

The  purpose  of  the  following  sections  is  to  provide  guidance  on  how  to  utilize  simulation  to  train  decision¬ 
making  under  stress.  The  simulation-based  training  approach  is  designed  specifically  to  enhance  small 
unit  leader  DM  skills  and  the  resilience  of  these  skills  to  stress.  There  are  three  key  aspects  of  this 
methodology  including: 

•  Effective  scenario  design  to  ensure  that  training  advances  DM  skills  relevant  to  target  training 
objectives  and  can  effectively  induce  stress, 

•  Integration  of  objective  and  quantitative  measures  of  DM  performance  at  the  process  level  and 
measures  of  associated  stress  response, 

•  Incorporation  of  learning  strategies  that  have  been  validated  to  improve  decision-making 
performance  and  bolster  a  trainee’s  response  to  stress. 

3.4.1  Decision  Events 

The  first  step  in  achieving  this  is  identification  of  target  training  objectives.  Analysis  of  training 
documentation  such  as  the  POI  or  training  manuals  typically  reveals  the  training  goals  of  the  course  such 
as  the  types  of  missions  and  TTPs  to  be  trained.  However,  in  cases  in  which  the  documentation  does  not 
provide  these,  additional  analysis  may  be  required  to  effectively  identify  the  training  goals  (see  Milham, 
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Carroll,  Stanney  &  Becker  (2008)  for  detailed  discussion  of  training  needs  analysis).  For  instance,  the 
ISULC  POI  identifies  a  set  of  T&R  standards  (e.g.,  lead  a  squad  on  a  patrol)  to  be  targeted.  This  provides 
a  framework  and  scope  for  the  types  of  decisions  to  incorporate  into  the  scenarios.  The  second  step  is  to 
identify  a  representative  set  of  relevant  decisions.  Through  course  observation  (e.g.,  what  are  the 
decisions  embedded  into  training  events),  training  content  review  (e.g.,  what  are  the  decision  anecdotes, 
and  situations  and  environments  incorporated)  and  SME  interviews  (e.g.,  descriptions  of  decisions  they 
have  made  in  past  deployments  and  surrounding  circumstances  and  outcomes),  a  set  of  relevant  and 
realistic  small  unit  leader  decisions  that  are  relevant  to  the  training  objectives  can  be  identified.  This 
supports  the  development  of  a  library  of  decision  types,  circumstances,  and  environments  from  which  to 
pull  scenario  decision  events  and  decision  factors  to  embed  in  the  environment.  The  third  step  is  scenario 
storyboarding  in  which  a  high  level  mission  description  (e.g.,  Area  of  Operation  (AO),  type  of  Military 
Operation,  etc.)  is  created  and  scenario  decisions  are  selected  and  tied  together  with  a  realistic  and 
engaging  narrative.  Each  scenario  is  designed  using  an  event-based  methodology  (Fowlkes  et  al.,  1998) 
in  which  five  decision  events  are  incorporated  to  provide  five  opportunities  for  a  trainee  to  practice 
decision  skills  as  well  as  five  opportunities  to  assess  decision-making  at  the  process  levels.  Each  of  the 
five  discrete  decision  events  build  upon  each  other  and  can  have  either  positive  or  negative 
consequences  for  the  participant  later  in  the  scenario,  based  on  the  quality  of  the  decision  that  was  made. 
The  scenarios  are  designed  such  that  DM  performance  impacts  mission  outcome.  Therefore,  the  more 
effective  the  DM  performance,  the  more  likely  the  trainee  is  to  succeed  in  their  mission.  These  events 
then  act  as  an  outline  for  the  full  mission  narrative  to  be  created.  Working  in  hand  with  SMEs,  a  narrative 
is  interwoven  around  these  events.  Each  scenario  is  designed  to  maximize  decision  opportunities  in  a 
short  period  of  time  with  situations  unfolding  very  quickly  and  each  scenario  taking  approximately  15-30 
minutes  to  complete. 

3.4.2  Stress  Analysis 

Contextually-relevant  simulation-based  stress  induction  techniques  (e.g.,  dead  bodies,  artillery  fire;  see 
Bouchard  et  al.,  2012)  must  be  integrated  to  induce  stress.  Research  to  date  has  found  that  stressors 
such  as  seeing  dead  bodies,  receiving  artillery  or  small  arms  fire,  knowing  someone  is  seriously  injured  or 
killed  are  stressful  combat  experiences  that  translate  into  simulation-based  stressors  because  they  are 
frequent,  strong  psychological  challenges  that  last  long  enough  to  be  used  in  narratives  (Bouchard,  Baus, 
Bernier,  &  McCreary,  2010).  Key  aspects  of  these  stressors  which  have  been  shown  to  impact  the  level  of 
stress  response  include  unpredictability,  novelty  and  lack  of  control  (Dickerson  &  Kemeny,  2004).  The 
scenarios  incorporate  stressors  such  as  these  in  the  narratives  to  induce  varying  levels  of  stress.  An 
attempt  is  made  to  vary  stress  levels  induced  by  the  scenarios  by  varying  the  number,  severity  and 
duration  of  stressors  encountered  throughout  the  scenario  (e.g.,  taking  almost  continuous  gunfire  or 
artillery  rounds  from  various  locations  out  of  sight).  Based  on  previous  research  and  the  results  of  the 
STAR-DM  effort,  the  following  is  a  set  of  recommendations  for  incorporating  stressors  into  training  for 
Warfighters: 

1.  Incorporate  socio-evaluative  stress.  Where  possible,  incorporate  opportunities  for  an  individual  to 
feel  he  is  being  judged  by  a  group  of  experts.  Those  acting  as  assessors/judges  should  attempt  to 
maintain  flat  affect,  maintain  eye  contact,  minimize  nodding  and  provide  pointed  feedback  when 
possible.  This  can  be  done  external  to  training  scenarios,  or  within  a  virtual  environment. 

a.  Example(s):  During  classroom,  virtual  or  live  training  exercises,  require  the  trainee  to  publicly 
(verbally)  brief  a  plan  or  perform  a  task.  Ensure  trainee  is  aware  of  performance  being  judged 
by  expert  instructors  and  peers  and  provide  public  and  pointed  feedback. 

2.  Incorporate  uncontrollability.  When  designing  live  or  virtual  training  scenarios,  incorporate  situations 
in  which  any  action  taken  by  the  trainee  does  not  seem  to  improve  the  situation. 

a.  Example(s):  Provide  trainee  with  faulty  weapon  or  tool  which  malfunctions  when  needed,  or 
with  little  ammunition  per  clip,  such  that  the  trainee  must  reload  often.  Incorporate  enemies  with 
low  vulnerability  that  are  virtually  impossible  to  overcome. 

3.  Incorporate  unpredictability.  When  designing  live  or  virtual  training  scenarios,  incorporate  unexpected 
situations  for  which  the  trainee  likely  did  not  have  a  contingency. 
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a.  Example(s):  Incorporate  highly  unlikely  enemy  weapons,  capabilities,  or  tactics.  Incorporate 
situations  that  could  not  be  anticipated  based  on  the  intelligence  provided  in  the  mission  order, 
such  as  the  presence  of  indirect  fire  after  being  told  that  the  enemy  did  not  have  that  capability. 
4.  Incorporate  novelty.  When  designing  live  or  virtual  training  scenarios,  incorporate  situations  with 
which  trainees  have  little  to  no  previous  experience, 
a.  Example:  Incorporate  terrain  features  with  which  trainees  have  little  experience  or  give  enemies 
advanced  weapons  systems  that  trainees  have  no  experience  responding  to.  Incorporate 
enemy  tactics  and  Rules  of  Engagement  different  than  those  used  in  recent  AOs. 

Each  scenario  is  designed  with  five  events  that  require  some  action  on  the  part  of  the  Marine  Squad 
Leader,  have  clear  decision  alternatives  upon  which  decision-making  can  be  measured,  and  are 
increasing  in  stress  levels  based  on  a  stress  analysis  conducted  by  the  researchers.  The  following  stress 
analysis  example  is  based  on  an  Experiment  2  scenario  titled  "Sleight  of  Hand."  The  participant  is  tasked 
with  retrieving  explosives  from  a  white  van  at  night.  The  participant  is  told  before  the  mission  starts  not  to 
get  too  close  to  any  military  personnel  and  to  always  observe  enemies  first  before  taking  action.  When 
the  participant  comes  across  the  van,  he  finds  two  military  officers  near  the  van.  There  are  also  civilians 
walking  around  in  the  marketplace  where  the  van  is  located. 

Step  1  of  the  stress  analysis  is  to  identify  all  of  the  potential  stressors  in  the  event  from  a  predetermined 
list.  All  of  the  potential  stressors  are  identified  under  the  assumption  that  the  participant  correctly 
executes  his  tasking  during  the  event.  A  predefined  list  of  stressors  was  used  for  the  stress  analysis.  The 
stressors  applying  to  this  event  include: 

•  Emotional  -  hazard  (e.g.  user  not  briefed  that  enemy  soldiers  would  be  by  the  van) 

•  Cognitive  -  auditory  distraction  (e.g.  the  user  can  hear  the  civilians  talking) 

•  Cognitive  -  visual  distraction  (e.g.  the  user  can  see  civilians  walking  around  randomly) 

•  Cognitive  -  limited  visibility  (e.g.  the  user  is  performing  the  mission  at  night  time) 

•  Emotional  -  tactical  patience  (e.g.  the  user  must  wait  for  the  enemies  to  leave  the  area  where  the 
van  is  located) 

•  Cognitive  -  workload  (e.g.  the  user  must  retrieve  explosives  from  the  van) 

Once  all  of  the  stressors  are  identified,  they  are  rated  according  to  their  characteristics.  There  are  five 
characteristics  a  stressor  can  have: 

1 .  Duration  (how  long  the  stressor  is  present  during  the  event) 

2.  Intensity  (how  much  of  a  threat  the  stressor  poses  to  the  user) 

3.  Unpredictability  (how  well  the  threat  could  have  been  anticipated  by  the  user) 

4.  Uncontrollability  (how  much  the  user  can  do  to  remove  the  stressor  from  the  event  within  the 
context  of  the  mission) 

5.  Novelty  (how  much  familiarity  the  user  has  with  the  stressor) 

Points  are  awarded  to  each  of  these  characteristics:  for  example,  duration  can  assume  the  values  of  0, 
.25,  .5,  .75,  and  1.  Unpredictability  can  be  either  0  or  1.  Each  value  for  Intensity  -  Novelty  are  added  up 
and  multiplied  by  the  duration  value  for  each  stressor.  This  yields  a  total  value  for  an  individual  stressor, 
and  then  the  total  values  for  each  event  stressor  are  added  up  to  yield  an  event  stress  total.  The  event 
stress  totals  are  then  added  up  to  yield  a  scenario  stress  total.  The  events  are  designed  to  increase  in 
stress  levels  over  the  course  of  the  scenario. 

3.4.3  Assessment 

Each  scenario  incorporates  mission  performance  outcome  measures  to  assess  overall  performance,  such 
as  whether  the  mission  objectives  were  achieved,  time  to  complete  the  mission,  and  number  of  civilian 
and  friendly  casualties  resulting.  However,  to  effectively  improve  an  individual’s  DM  performance  under 
stressful  conditions,  it  is  necessary  to  assess  decision-making  at  the  process  level  (i.e.,  not  only  decision 
outcome,  but  whether  the  trainee  is  effectively  observing,  orienting  to  the  situation,  etc.),  including  the 
associated  stress  response  in  order  to  understand  where  and  why  breakdowns  in  the  decision-making 
cycle  occur.  This  ultimately  allows  for  remediation  to  be  tailored  to  the  individual. 
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3.4.3. 1  Decision-making  Assessment 

The  framework  for  assessing  DM  performance  is  based  on  the  OODA  Loop  (Boyd,  1987)  which 
decomposes  the  decision-making  process  into  four  key  stages:  1)  Observe  (collection  and  integration  of 
sensory  information),  2)  Orient  (interpretation  of  sensory  information  to  determine  the  current  environment 
and  situation),  3)  Decide  (evaluation  of  alternative  courses  of  actions  and  response  selection),  and  4)  Act 
(planning  and  execution  of  response).  Decomposition  of  the  DM  process  into  these  four  stages  allows 
decision-making  to  be  assessed  at  the  process  level  in  order  to  understand  the  root  cause  of  decision 
breakdowns.  So,  in  addition  to  typical  decision  outcome  measures  of  decision  effectiveness  and  time  to 
decide,  each  stage  of  the  decision  process  can  be  assessed  independently.  For  the  Observe  stage,  there 
are  critical  AOIs  which  should  be  attended  to  in  order  to  effectively  inform  a  situation.  The  Squad  Leader's 
observations  were  inferred  from  squad  communications  about  AOIs.  The  Orient  and  Decide  stages  are 
much  more  challenging  to  assess  as  these  are  for  the  most  part  internal  and  unobservable  processes.  A 
combination  of  two  techniques  is  used  to  assess  these  stages  of  the  DM  process.  First,  contextually 
relevant  online  queries  (Gugerty,  2011;  Klein  &  Floffman,  1992)  in  the  form  of  information  “pull”  from 
higher  command  are  incorporated  into  the  scenario  in  which  decision  makers  are  prompted  to  answer 
questions  related  to  the  events  unfolding  as  they  perform.  These  queries  are  designed  to  imitate  the 
process  of  radio  communication  with  higher  command  so  as  to  minimize  interference  with  task 
performance.  Second,  by  carefully  crafting  scenarios  designed  to  elicit  certain  responses,  inference- 
based  measures  (Gugerty,  2011)  are  utilized  to  infer  Orient  and  Decide  stages  based  on  monitoring 
communications  and  actions.  For  instance,  in  order  to  assess  the  Orient  stage,  at  various  points 
throughout  the  scenario  the  trainee  is  asked  to  verbally  report  what  they  see  and  what  it  means.  Based 
on  the  percentage  of  critical  situation  factors  reported,  orientation  effectiveness  level  is  assessed.  In  order 
to  assess  the  Decide  stage,  the  trainee  is  asked  to  verbally  report  the  possible  courses  of  action  they  can 
take.  Based  on  the  courses  of  action  reported  and  the  actions  taken,  the  effectiveness  of  the  decide  stage 
is  assessed.  The  Action  phase  is  assessed  based  on  how  effectively  the  response  is  carried  out; 
however,  Action  is  not  a  major  focus  of  this  effort  as  it  is  typically  an  indication  of  tactical/technical  skills, 
not  decision  effectiveness.  These  measures  facilitate  the  assessment  of  each  stage  independently  and 
provide  granularity  in  assessment  to  understand  where  breakdowns  in  the  DM  process  originated  and 
identify  patterns  of  these  breakdowns  across  decisions  (e.g.,  does  a  performer  have  issues  with  situation 
assessment  that  need  to  be  remediated). 

3.4.3. 2  Stress  Assessment 

Stress  can  impair  the  DM  process  leading  to  reductions  in  quality  of  and  confidence  in  decisions  (for 
review  of  impacts  of  stress  on  decision-making  see  Carroll  et  al.,  2012).  Assessment  of  an  individual’s 
stress  response  during  DM  performance  can  assist  in  determining  if  performance  decrements  are  due  to 
basic  skill  deficiencies  or  failure  of  the  skill  to  be  resilient  to  stress,  and  therefore  inform  how  to  remediate 
performance  decrements.  Additionally,  real-time  monitoring  of  stress  can  be  used  to  provide  biofeedback 
to  trainees  which  was  shown  to  be  effective  in  helping  them  moderate  their  stress  levels  resulting  in 
improved  performance  in  Experiment  2.  Stress  response  can  be  objectively  and  quantitatively  assessed, 
in  near  real  time,  without  disruption  to  task  performance  by  utilizing  physiological  measures.  When  an 
individual  encounters  a  significant  stressor,  the  sympathetic  or  “fight  or  flight”  division  of  the  autonomic 
nervous  system  increases  in  activity,  resulting  in  hormone  release  and  subsequent  physiological  effects 
on  multiple  organ  systems.  Among  a  myriad  of  effects,  heart  rate  increases,  pupils  dilate,  blood  vessels 
constrict,  and  sweat  glands  become  active. 

Existing  approaches  to  stress  detection  use  a  wide  array  of  features  calculated  from  sensor  data 
measuring  various  aspects  of  heartbeat  including  include  PPG  or  ECG  (Sun,  Kuo  et  al.  2010;  De  Santos, 
Sanchez-Avila  et  al.  2011 ;  Plarre,  Raij  et  al.  201 1),  skin  conductance  measurement  (Bakker,  Pechenizkiy 
et  al.  201 1 ;  Alamudun,  Choi  et  al.  2012;  Choi,  Ahmed  et  al.  201 2),  and  measurement  of  respiration,  all  of 
which  are  responsive  to  increased  sympathetic  nervous  system  activity  associated  with  stress  (Everly  and 
Lating  2002).  Standard  supervised  machine  learning  methods  have  been  used  previously  to  develop 
stress  classifiers,  which  require  subjects  to  engage  in  tasks  known  to  induce  stress  so  that  stress  or  non¬ 
stress  labels  can  be  assigned  to  the  input  features.  Previous  work  has  emphasized  the  difficulties 
imposed  on  stress  classification  by  individual  subject  variability  in  physiological  responses  to  stress  (De 
Santos,  Sanchez-Avila  et  al.  2011;  Alamudun,  Choi  et  al.  2012).  Another  concern  is  the  physical  activity 
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of  subjects,  which  triggers  similar  cardiovascular  and  electrodermal  physiological  signals  as  stress, 
leading  to  masking  and  confounds  of  stress  detection  (Sun,  Kuo  et  al.  2010;  Alamudun,  Choi  et  al.  2012). 
The  major  challenge  in  using  mobile  physiological  sensors  to  quantify  stress  is  the  lack  of  robust  and 
clinically  tested  algorithms  to  classify  stress  in  a  mobile  environment  in  real  time  (Martfnez-Perez,  de  la 
Torre-Dtez  et  al.  2013).  Previous  stress  monitoring  algorithms  have  been  built  with  traditional  laboratory 
physiological  sensor  suites  that  do  not  translate  well  to  operational  settings  (Plarre,  Raij  et  al.  2011; 
Alamudun,  Choi  et  al.  2012).  New  wearable  devices  with  clinical  grade  sensors  such  as  the  Empatica  E4 
which  sends  PPG,  EDA,  temperature  and  accelerometry  data  via  Bluetooth  LE  and  associated  algorithms 
such  as  the  Operational  Stress  Index  (OSI;  Dechmerowski  et  al.,  2014)  have  the  potential  to  take  real¬ 
time  stress  monitoring  outside  of  the  laboratory.  The  capability  of  assessing  physiological  metrics 
associated  with  stress  reactivity,  such  as  cardiovascular  and  electrodermal  activity,  in  context  via  sensor- 
based  assessment  of  movement  and  temperature,  along  with  algorithm  personalization  via  individual 
baselining,  allows  for  high  accuracy  stress  classification  in  an  ambulatory  environment.  Continuing  work 
with  the  Department  of  Homeland  Security  (DHS)  is  focusing  on  expanding  the  sensor  technology  used  to 
assess  stress,  including  MClO’s  BioStamp  RC,  and  Physical  Optics  Corporation’s  (POC)  MVSS  sensor 
suite,  in  development  for  the  Air  Force  Research  Laboratory.  Additional  work  with  DHS  is  focused  on 
developing  a  visualization  tool  to  view  multiple  trainee’s  stress  state  in  real  time,  and  refining  the  classifier 
to  function  in  extreme  environments,  such  as  exposure  to  high  temperatures  and  humidity. 

3.4.4  Incorporated  Learning  Strategies 

There  are  innumerable  learning  strategies  which  have  been  explored  in  the  laboratory  and  the  field  and 
have  shown  varying  degrees  of  success  at  effectively  training  decision-making  and/or  stress  skills  (e.g., 
Klein,  1993;  Lipshitz,  Klein,  Orasanu,  Salas,  2001;  Batha  &  Carroll  2007;  Calderon  &  Thompson,  2004). 
Pre-training  exercises,  such  as  planning  exercises  which  require  trainees  to  identify  and  layout  the  plan 
for  how  they  will  accomplish  a  goal  have  been  effectively  utilized  (Blackburn  et  al.,  2004).  A  key  learning 
strategy  which  can  be  utilized  during  training  is  an  Event-Based  Approach  to  Training  (EBAT;  Fowlkes  et 
al.,  1998)  in  which  training  opportunities  are  systematically  created  by  presenting  events  designed  to  elicit 
specific  skills  or  behaviors  (e.g.,  decision-making).  EBAT  has  been  shown  effective  at  training  DM  skills 
across  a  range  of  domains  (Fowlkes  et  al.,  1998).  After  action  review  techniques  such  as  error-based  or 
causal-based  feedback  (Carroll  et  al.,  2008,  Carroll,  2010)  provide  the  opportunity  to  target  decision¬ 
making  at  the  process  level.  Errors  are  considered  to  be  valuable  opportunities  to  clarify 
misunderstandings  in  learners  (Mory,  2004)  and  the  identification  of  underlying  causes  of  deficient 
processes  allows  instructors  to  provide  meaningful  feedback  to  correct  these  deficiencies  (Salas  et  al., 
2007).  This  can  be  combined  with  specific  learning  strategies  such  as  attribute  isolation  in  which  central 
attributes  of  target  concepts  (e.g.,  why  cues  are  critical  for  a  situation  or  indicative  of  an  impending  threat) 
are  highlighted  to  improve  general  understanding  of  phenomenon  (Mason  &  Bruning,  2001). 

There  are  also  multiple  learning  strategies  that  can  be  used  to  target  an  individual’s  stress  response 
during  decision-making  training.  Biofeedback  is  a  process  by  which  individuals  learn  to  control  certain 
autonomic  nervous  system  functions  such  as  blood  pressure,  salivation,  sweat  gland  activity,  and  cardiac 
activity  through  feedback  signals  from  sensors  monitoring  physiological  responses  (Calderon  & 
Thompson,  2004).  BioFeedback  Training  (BFT)  techniques  typically  encompass  three  stages  wherein  a 
trainee  1)  acquires  awareness  of  maladaptive  physiological  responses,  2)  learns  to  control  the  response 
utilizing  techniques  such  as  deep  breathing  and  muscle  relaxation  and  3)  learns  to  transfer  this  control  to 
everyday  life  (Lehrer  &  Wolfolk,  1993).  As  shown  in  Experiment  2,  the  use  of  biofeedback  led  to  reduced 
stress  levels  and  potential  performance  increases.  Stress  exposure  strategies  such  as  Stress  Inoculation 
Training  (SIT;  Saunders  et  al.,  1996)  and  Stress  Exposure  Training  (SET;  Driskell  and  Johnston,  1998) 
also  incorporate  three  phases,  except  in  this  case  they  involve  1)  an  education  phase  to  help  the  trainee 
better  understand  the  nature  of  stress  and  stress  effects,  2)  a  skill  acquisition  and  rehearsal  phase  to 
facilitate  development  and  practice  a  repertoire  of  coping  skills  and  3)  an  application  phase  in  which 
coping  skills  are  applied  in  conditions  that  increasingly  approximate  the  transfer  environment.  Both 
techniques  have  been  shown  effective  in  increasing  a  trainee’s  ability  to  cope  with  stress  (Sheehy  & 
Horan,  2004;  Saunders,  Driskell,  Hall,  &  Salas,  1996;  Meichenbaum  &  Deffenbacher,  1988). 
Metacognition  strategies  can  be  utilized  to  facilitate  awareness  of  how  one  perceives  and  thinks  about 
stressors  as  well  as  an  accurate  appraisal  of  a  stressor  by  assessing  best,  worst,  and  most  likely 
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outcomes  (Narayanan,  2009).  Further,  metacognitive  strategies  can  also  be  utilized  to  increase 
awareness  of  the  detrimental  impact  the  stressor  is  having  on  an  individual’s  DM  process  (e.g.,  narrowing 
the  perceptive  field). 

3.5  Integrated  Simulation-based  Training  Approach 

The  above  presented  learning  strategies  are  all  feasible  to  implement  in  a  military  schoolhouse  setting. 
However,  the  challenge  remains  how  to  seamlessly  integrate  the  strategies  into  current  training 
curriculum  in  an  easy-to-use  simulation-based  approach.  To  achieve  this,  the  “Do  Something” 
instructional  approach  was  utilized  (DeVore,  2010).  The  “Do  Something”  approach  is  a  concise 
instructional  method  that  incorporates  four  components  in  which  1)  the  mission  is  presented  via  a  video 
narrative,  2)  the  trainee  is  given  a  brief  period  to  create  and  brief  their  plan,  3)  the  plan  is  executed  in  a 
simulation-based  scenario  and  4)  an  instructor  and  peer-based  after  action  review  is  conducted  utilizing  a 
series  of  tools  (e.g.,  performance  replay).  This  approach  provides  a  framework  for  anchoring  the  pre, 
during  and  post  training  strategies  referenced  above  into  a  simulation  based  training  package. 

First,  a  video  narrative  presents  the  orientation,  situation,  mission  objective  and  tactical  dilemma.  The 
video  provides  an  outline  of  the  mission  goals  and  cues  the  trainees  to  critical  factors  to  which  they 
should  attend  (Advance  Organizer).  This  is  delivered  in  a  realistic  and  immersive  video  designed  to 
increase  engagement  and  motivation  (Motivation  Strategy). 

Second,  the  trainee  performs  a  planning  exercise  in  which  they  analyze  the  situation,  devise  a  mission 
plan  and  develop  and  deliver  orders  to  their  team.  The  instructor  then  alerts  the  trainee  of  an  incorrect 
situation  assessment  or  failure  point  of  the  plan  (Crystal  Ball,  Premortem  Exercise).  The  trainee  must  re¬ 
plan  and  re-issue  tasks  to  the  team.  This  is  conducted  under  time  constraints  and  in  the  presence  of  unit 
leadership  to  induce  stress.  This  is  a  form  of  social-evaluative  stress,  which  can  be  induced  when  others 
negatively  judge  ones  performance  and  has  been  found  to  elicit  significant  and  reliable  physiological 
stress  responses  (Dickerson  &  Kemeny,  2004).  During  the  training,  the  trainee  is  equipped  with  a  limited 
set  of  invasive  physiological  sensors  (e.g.,  heart  rate,  EDA)  whose  output  is  displayed  for  the  instructor 
and  trainee  to  see  (BioFeedback).  As  the  plan  is  briefed,  the  instructor  can  point  out  to  the  trainee  times 
their  physiological  state  is  elevated,  the  resulting  physiological  response  (e.g.,  HR  increase,  sweat 
present,  tense  shoulders)  and  present  one  or  two  quick  and  easy  coping  strategies  to  reduce  stress 
response  (i.e.,  deep  breathing,  muscle  relaxation)  which  the  trainee  can  practice  and  is  then  encouraged 
to  employ  at  any  time  during  the  training  when  they  recognize  the  physiological  response  to  stress  they 
just  learned  about. 

Third,  after  the  plan  has  been  briefed,  the  trainee  leads  his  team  through  the  execution  of  the  plan  in  a 
simulation-based  scenario  designed  to  elicit  adaptive  decision-making  skills  under  stress.  The  trainee 
encounters  five  decision  events  as  he  performs  the  mission,  requiring  him  to  effectively  observe,  orient, 
decide  and  act  (event-based  training)  while  process  level  DM  performance  is  assessed  via  measures 
discussed  above.  During  scenario  execution,  the  trainee  experiences  a  series  of  simulation-based 
stressors  such  as  limited  visual  perception  (night  missions),  sudden  noise  exposure,  equipment  failures, 
and  receiving  enemy  fire,  as  well  as  cognitive  stressors  (e.g.,  time  pressure)  and  emotion  induction 
procedures  (e.g.,  dead  combatants,  soldiers  and  civilians).  The  scenarios  are  high  fidelity,  include  a 
variety  of  novel,  unexpected  challenges  and  increase  in  complexity  as  training  progresses. 

Fourth,  an  AAR  is  conducted  utilizing  a  series  of  tools.  While  mission  outcome  performance  is  briefly 
reviewed,  the  AAR  focuses  on  the  process  level  DM  performance  metrics,  emphasizing  the  root  cause  of 
decision  breakdowns  which  propagated  to  mission  failure  (e.g.,  consistent  failure  to  detect  critical  cues; 
mastery  orientation,  error-based,  causal-based  feedback).  For  errors  identified,  central  attributes  of  target 
concepts  (e.g.,  why  decision  alternative  selected  was  a  poor  choice)  are  highlighted  to  improve  decision 
skills  (attribute  isolation). 
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4.0  Squad  Leader  Training  Packages  (SLTPs) 

The  training  packages  are  comprised  of  the  following  components: 

(1)  Multimedia  Videos.  VBS2-based  Road  to  War  and  Operational  Update  videos  detail  the  situation 
and  environment  and  are  designed  to  engage  and  motivate  the  trainees. 

(2)  Mission  Orders/  Fragmentary  Orders  (FRAGOs).  Paper-based  and/or  video-based  mission 
orders/FRAGOs  are  included  for  delivery  by  the  instructor/facilitator  for  viewing  by  the  trainees.  After 
receiving  mission  orders,  the  packages  are  designed  to  allow  the  Squad  Leader  to  plan  the  mission  and 
task  fire  teams  based  on  these  orders.  There  are  guidelines  provided  to  make  this  a  more  stressful 
experience,  such  as  time  pressure  and  instructor  maintaining  eye  contact  and  providing  pointed 
feedback. 

(3)  VBS2  Scenarios.  Each  package  contains  four  VBS2  scenarios  designed  to  allow  the  entire 
squad  to  execute  the  Squad  Leader’s  plan  in  a  networked  VBS2  simulation.  As  the  team  performs  the 
scenario,  they  encounter  five  discrete  decision  events  in  which  the  Squad  Leader  has  to  determine  the 
best  course  of  action  and/or  adapt  his  plan. 

(4)  Measurement  and  Debrief  Tools.  The  packages  have  integrated  paper-based  assessment  tools 
that  allow  instructors  to  assess  performance  at  the  process  level.  The  assessment  tools  are  designed  to 
also  facilitate  a  DM  focused  debrief. 

(5)  Instructor  Preparation  Guides.  Each  package  is  accompanied  by  an  IPG,  which  walks  an 
instructor  step-by-step  through  delivery  of  the  training  packages. 

Detailed  descriptions  of  each  of  the  four  SLTPs  are  provided  in  the  following  sections. 

4.1  SLTP  1  -  Genocide  Intervention 

A  drug  kingpin  named  Sanchi  Abacca  has  fabricated  a  small  army  in  central  Africa  known  as  the  African 
Revolutionary  Melee  (ARM).  His  goal  is  to  kill  all  people  who  impede  the  future  economic  growth  of  Africa. 
The  26th  Marine  Expeditionary  Unit  has  been  ordered  to  conduct  operations  to  disrupt  the  ARM  activities, 
because  it  is  feared  that  any  delay  will  give  the  ARM  an  opportunity  to  employ  weapons  of  mass 
destruction. 

4.1.1  Scenario  1  -  Maritime  Assault 

This  scenario  will  challenge  the  squad  to  correctly  prioritize  enemy  boats  in  an  ambush.  After  the 
ambush,  the  squad  will  have  to  effectively  combat  foot  mobiles,  mortar  attacks  and  enemy  snipers.  This 
scenario  addresses  the  following  T&R  Objectives:  031 1 -OFF-2002,  INF-MAN-4301,  and  INF-MAN-4002. 
Table  9  displays  the  events  contained  in  the  Maritime  Assault  scenario. 


Table  9.  Maritime  Assault  Scenario  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Woman 

Screaming 

Marines  begin  movement  from  insert 
landing  zone  (LZ)  towards  ambush 
position  and  they  hear  women 
screams  and  gunshots  about  300-400 
m  north  of  their  position. 

•  Send  full  squad  to  investigate 

•  Send  part  of  squad  to  investigate 

•  Have  full  squad  continue  to 
ambush  position 
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2 

Static  Enemy 
Technical 

Marines  are  continuing  and  spot  an 
enemy  jeep  as  they  are  approaching 
the  river  with  2  ARM  soldiers  in  it 
(soldiers’  backs  are  to  the  Marines). 

•  Squad  on  line,  assault  through 
enemy  forces 

•  Coordinate  maneuver  with 
minimal  Marine  shooters  and 
minimal  force  required 

•  Bypass  the  jeep  (in  which  case, 
soldiers  will  attack  Marines  as 
soon  as  they  go  around  them) 

3 

Enemy 

Watercrafts 

Marines  have  set  into  their  ambush 
position,  and  then  3  boats  of  various 
sizes  appear  on  the  expected  route 
with  5-6  uniformed  enemies  per  boat. 

All  3  boats  have  large  weapon 
systems  on  them,  but  one  boat  is 
larger  with  communication  equipment 
also. 

•  Prioritize  the  larger 
communications  boat  as  highest 
priority 

•  Prioritize  the  lead  boat  as  the 
highest  priority 

•  Make  the  boats  equal  priority 

4 

Enemy 

Squad 

About  2-4  minutes  after  ambush  is 
over  (either  all  boats  are  destroyed  or 
any  of  the  boats  get  away),  a  squad¬ 
sized  enemy  unit  will  move  in  from  the 
West. 

•  Stand  their  ground  and  fight 
enemy  squad 

•  Egress 

•  Marines  attempt  to  detach  a  fire 
team  in  order  to  establish  an  L- 
shaped  attack  (i.e.  flanking 
enfilade) 

5 

Sniper/Mortar 

Attack 

When  Marines  are  about  200  m  from 
the  town,  2-3  mortars  impact  near 
them,  then  about  30  sec  later  2  enemy 
snipers  will  begin  engaging  the 

Marines  from  2  separate  shooting 
positions.  An  enemy  technical  vehicle 
will  appear  from  the  East  and  engage. 

•  Engage  sniper(s) 

•  Engage  the  technical  vehicle 

•  Execute  bounding  maneuver 
towards  the  tree  line  (to  the  rear) 
while  suppressing  technical 

•  Mask  movement  with  smoke  and 
egress 

4.1.2  Scenario  2  -  Cutting  Off  an  Enemy  Assault 

This  scenario  will  have  the  squad  waiting  to  ambush  an  enemy  convoy  with  a  combination  of  indirect  and 
direct  fire.  They  will  observe  several  groups  that  do  not  match  the  enemy  description,  and  they  will  be 
forced  to  make  shoot/no-shoot  decisions.  Finally  the  target  enemy  group  will  appear,  and  the  squad  is 
expected  to  engage  them  appropriately.  This  scenario  addresses  the  following  T&R  Objectives:  0311- 
DEF-2002,  INF-MAN-4002.  Table  10  displays  the  events  contained  in  the  Cutting  Off  an  Enemy  Assault 
scenario. 


Table  10.  Cutting  Off  an  Enemy  Assault  Scenario  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

10  Civilians 

10  civilians  (3  with 
weapons)  run  past  the 
hill  from  E  to  W 

•  Order  Squad  to  engage 

•  Utilize  indirect  fire 

•  Identify  the  group  as  civilians  and  let  them 
go 

2 

Civilian  Vehicles 

2  civilian  vehicles  begin 
driving  towards  Marines’ 
position  past  one  on-call 
target  location 

•  Use  squad's  weapons  to  shoot  at  vehicles 

•  Use  indirect  fire  to  shoot  at  vehicles 

•  Identify  the  vehicles  as  civilian  and  monitor 
them  from  their  position 
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3 

2  Enemy 
Technicals 

2  enemy  technical 
vehicles  will  come  from 
the  NE,  along  the 
enemy's  expected  path 

•  Use  indirect  fire  to  shoot  vehicles 

•  Use  squad's  weapons  to  shoot  vehicles 

•  Allow  the  vehicles  to  go  through  and  see 
what  they  do  or  allow  the  rest  of  Alpha 
Company  to  deal  with  them 

4 

3  Squad  Foot 
Mobiles 

3  squads  of  foot  mobile 
enemies  come  down  the 
expected  path  from  the 

NE 

•  Initiate  contact  with  indirect  fire  asset 
followed  by  squad's  weapons 

•  Initiate  contact  with  squad's  weapons 
using  indirect  fire  asset  as  defensive 
measure 

•  Only  use  squad's  weapons  against  the 
enemy  element 

•  Call  for  fire  and  egress 

•  Displace  squad 

5 

6  Technical 
Vehicles 

6  technical  vehicles 
come  down  the 
expected  path  from  the 

NE 

•  Immediately  break  contact  to  get  cover 
and  concealment 

•  Hold  off  enemy  until  they  can  utilize 
another  indirect  fire  attack 

•  Use  indirect  fire  to  suppress  while  they 
break  contact  and  get  cover  and 
concealment 

4.1.3  Scenario  3  -  Attacking  an  ARM  Compound 

This  scenario  will  have  the  squad  attacking  an  ARM  compound  to  perform  Tactical  Site  Exploitation 
(TSE).  The  squad  will  face  a  variety  of  foot-mobile  and  vehicle-borne  enemies.  This  scenario  addresses 
the  following  T&R  Objectives:  031 1-DEF-2002,  INF-INT-4001,  INF-MAN-6004,  and  INF-FSPT-4001 . 
Table  1 1  displays  the  events  contained  in  the  Attacking  an  ARM  Compound  scenario. 

Table  1 1.  Attacking  an  ARM  Compound  Scenario  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Civilian  & 

Guard  Patrol 

Squad  is  approaching 
compound  and  a  2-man  ARM 
patrol  with  a  civilian  prisoner 
walks  by  in  their  field  of  view 
approximately  200  m  from  the 
compound. 

•  Shoot  the  ARM  soldiers  or  take 

cover 

•  Watch  where  they  go  and  try  to 
conduct  pattern  analysis 

•  Attempt  to  circumvent  ARM  soldiers 

2 

Death  Squad 

5-ton  arrives  at  the  front  gate  of 
the  compound  with  2  ARM 
soldiers  and  5  civilians. 

Civilians  line  up  against  the  wall 
and  one  of  the  soldiers  begins 
loading  his  weapon. 

•  Attack  ARM  soldiers  before  they  can 
execute  civilians 

•  Attack  soldiers  immediately  after 
they  execute  the  civilians 

•  Remain  covered  and  continue  with 
planned  attack 

3 

2  Enemy 
Technicals 

Sporadic  gunfire  is  heard  off  in 
the  distance.  2  enemy  technical 
vehicles  will  be  passing  by 
behind  the  Marines  about  200 
m  away  and  go  static.  After  30 
sec  they  will  drive  to  the 
compound. 

•  Engage  the  vehicles  while  they  are 
moving  toward  the  static  position 

•  Wait  to  see  what  happens  and  then 
engage  them  within  30  seconds  after 
they  are  static 

•  Attempt  to  engage  them  once  they 
start  moving  towards  the  compound 

4 

15  ARM 

Soldiers  in  Tree 
Line 

After  TSE,  1 5  ARM  soldiers  will 
come  in  from  the  tree  line 
armed  with  RPGs  and  medium 
machineguns 

•  Bound  and  break  contact 

•  Stay  and  fight 
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5  BMP  at  LZ  Marines  see  a  BMP  about  500 

•  Call  for  extraction  for  the  secondary 

m  from  their  location  and  about 

LZ 

350  m  from  the  LZ.  After  3 

•  Call  for  extraction  for  the  primary  LZ 

minutes  of  static  behavior,  the 

ZSU-23-4  will  begin  to  move 

towards  Marines  and  engage 

them. 

4.1.4  Scenario  4  -  Evacuation 

This  scenario  will  have  the  squad  facilitating  an  evacuation  of  civilians.  They  will  witness  a  friendly 
helicopter  crash,  and  have  to  decide  how  best  to  help  the  surviving  pilots  defend  themselves  against 
enemy  foot  mobiles.  This  scenario  addresses  the  following  T&R  Objectives:  0300-PAT-1009  and  INF- 
MAN-4301.  Table  12  displays  the  events  contained  in  the  Evacuation  scenario. 

Table  12.  Evacuation  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Abandoned 

Technical 

Squad  finds  an  enemy  vehicle  with  a 
heavy  machinegun.  The  vehicle  has 
broken  down  and  there  are  2  dead 
enemies  on  the  ground  next  to  it,  but 
the  weapon  still  has  ammunition. 

•  Have  someone  on  the  squad 
mount  the  enemy 
machinegun  and  use  it 

•  Do  not  have  anyone  mount  it 

2 

Civilians 

8  civilians  (2  are  armed,  all  male) 
come  running  from  the  tree  line  into 
the  sector  of  the  squad  towards  the 
town 

•  Shoot  them 

•  Do  not  shoot  them 

3 

Helo  Crash 

A  helo  crashes  in  front  them  about 

1 000  m  away 

•  Send  all  or  part  of  squad  to 
check  for  survivors 

•  Call  Higher;  request  guidance 

4 

Mortar  Brackets 

1  ilium  round  goes  off.  1  mortar  hits 
200  m  in  front  of  squad.  1  mortar 
round  hits  100  m  behind  the  squad. 

1  mortar  round  lands  5  m  from  the 
enemy  vehicle.  1  large  explosion 
and  several  smaller  explosions 
occur  about  600  m  in  front  of  them  in 
the  tree  line  &  remnants  of  the 
mortar  section  that  fired  upon  them 
are  visible. 

•  Move  the  squad  out  of  the 
current  fighting  position  (more 
than  1 00  m  away) 

•  Keep  them  in  relatively  the 
same  position  (within  100  m) 

5 

Friendly  Fire 

Fight 

Marines  will  hear  both  M-1 6  and  AK- 
47  weapons  engaging  in  a  firefight, 
implying  that  the  fight  is  friendly  vs. 
enemy.  Initially  Marines  will  only  be 
able  to  see  some  tracer  rounds  in 
addition  to  hearing  the  weapons. 

The  friendly  pilot  crew  will  ascend  up 
the  hill  towards  the  Marines.  Enemy 
tracer  rounds  will  be  impacting 
around  the  pilot  crew.  Some 
enemies  will  become  visible  at  the 
cusp  of  the  tree  line. 

•  Suppress  the  enemies  over 
the  head  of  the  pilot  crew 

•  Move  the  squad  to  a  position 
where  they  can  suppress  the 
enemy  without  firing  over  the 
heads  of  the  pilot  crew 

•  Do  not  suppress  the  enemy 
to  avoid  shooting  over  the 
heads  of  the  pilot  crew 
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4.2  SLTP  2  -  Interior  Collapse 

In  the  fictitious  country  of  Sahrani,  a  civil  war  is  raging.  One  side,  the  People’s  Sahrani  Army  (PSA)  has 
made  threats  towards  the  US  and  other  neighboring  countries  if  they  intervene  in  Sahrani's  domestic 
dispute.  The  US  has  sided  with  the  PSA's  opposition,  the  Cortez  Parliamentary  Party  (CPP)  due  to  their 
attitudes  toward  preserving  international  tourism  in  the  country  and  their  measured  and  logical  approach 
to  the  mining  of  Promethium,  an  element  that  can  be  used  in  the  creation  of  nuclear  weapons.  The 
Marines  of  the  22nd  Marine  Expeditionary  Unit  are  being  deployed  to  support  the  CPP. 

4.2.1  -  Amphibious  Assault 

This  scenario  will  challenge  the  squad  to  clear  a  town  and  gain  control  of  a  prominent  hill  in  the  area.  The 
squad  will  face  multiple  types  of  adversaries  (e.g.  foot  mobiles,  snipers,  technicals,  and  full  5-tons).  This 
scenario  addresses  the  following  T&R  Objective:  INF-MAN-4001.  Table  13  displays  the  events  contained 
in  the  Amphibious  Assault  scenario. 


Table  13.  Amphibious  Assault  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Beach 

Explosions 

Marines  start  dismounted  from  the 
Amphibious  Assault  Vehicle  (AAV). 
After  squad  has  moved  about  halfway 
between  AAV  and  town,  they  will  hear 
and  see  explosions  west  of  their 
objective  (Parato). 

•  Disperse 

•  Stay  together 

•  Stay  together;  Update  Higher 

2 

Civilians 
Murdered  by 
Enemies 

As  Marines  approach  Parato,  a  small 
group  of  civilians  (5-6)  attempt  to  flee 
the  town.  Gunshots  are  heard  as  the 
civilians  flee  on  foot  &  some  civilians 
are  shot  and  killed.  As  the  Marines 
continue  their  movement,  they  will  see 

2  enemies  flee  towards  the  east. 

•  Engage  enemy  while  remaining 
static  (pursue  by  fire) 

•  Engage  while  pursuing  enemy 

•  Do  not  shoot  enemy  at  all 

3 

Civilian  Car 

Civilian  car  comes  speeding  towards 
Marines  from  direction  in  which 
enemy  uniformed  personnel  just  ran. 
At  the  same  time,  gun  shots  are  heard 
to  the  north  (left)  of  the  squad. 

•  Engage  vehicle 

•  Not  engage  vehicle 

4 

Enemy 

Snipers 

As  the  Marines  approach  the  LOA,  a 
six  man  sniper  team  will  open  fire  on 
the  Marines  from  the  top  of  the  hill. 

•  Ignore  sniper  while  they  continue 
clearing  town 

•  Stop  clearing  the  town,  go  firm, 
report  to  higher,  prosecute 
afterwards 

•  Split  up  with  part  of  squad  going 
after  sniper  and  part  of  squad 
finish  clearing  town 

5 

Sniper/Mortar 

Attack 

Once  the  Marines  have  obtained  the 
hill  position,  enemy  reinforcements 
will  begin  to  appear  from  the  north  in 
the  form  of  an  enemy  5-ton  vehicle 
deploying  a  squad  and  two  enemy 
technicals  providing  support  by  fire  for 
the  advancing  enemy  squad. 

•  Consolidate  all  Marines  to  single 
linear  firing  position. 

•  Spread  fire  teams  out  and  protect 
by  fire. 

•  Break  contact. 
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4.2.2  -  Helo  Raid 

This  scenario  will  have  the  squad  infiltrating  an  enemy  compound  to  destroy  key  radar  equipment.  The 
squad  will  face  decisions  about  whether  or  not  to  remain  concealed  and  undetected  and  when  to  destroy 
the  targets.  This  scenario  addresses  the  following  T&R  Objectives:  0311 -OFF-2003  and  INF-ASLT-3006. 
Table  14  displays  the  events  contained  in  the  Helo  Raid  scenario. 

Table  14.  Helo  Raid  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Ilium  Round 

After  a  few  minutes  of  movement 
towards  the  objective  compound, 
an  air  illumination  round  goes  off 
near  squad. 

•  Stop  and  wait  for  illumination  round  to 
dissipate 

•  Continue  moving  toward  their 
objective 

2 

Civilian  Man 

As  Marines  continue  their  approach 
toward  the  objective,  a  random 
civilian  appears  and  walks  by  near 
the  squad. 

•  Shoot  the  civilian 

•  Continue  movement  ignoring  civilian's 
presence 

•  Wait  and  hide  until  the  civilian  passes 

3 

Civilian 

Execution 

Civilian  pickup  truck  approaches 
from  the  north  and  stops  between 
the  Marines  and  their  objective. 

The  Marines  will  witness  two 
uniformed  enemy  dismount  the 
vehicle  with  a  civilian.  One  of  the 
enemy  soldiers  will  execute  the 
civilian. 

•  Hide  and  wait  until  vehicle  leaves 

•  Prosecute  the  uniformed  guys 
(resulting  in  increased  enemies, 
actively  searching) 

•  Try  to  circumvent  the  vehicle 

4 

Radar 

Equipment 

Marines  make  it  to  the  compound 
and  begin  searching  for  the 
equipment.  In  the  north  portion  of 
the  compound  they  will  find  2  (or  4) 
OPFOR  with  weapons  guarding  the 
equipment.  There  will  also  be  dead 
civilians  around  the  equipment. 

•  Set  off  charge  immediately  (potentially 
compromising  the  squad’s  position 
when  you  still  need  to  move  to  the 
extraction  point) 

•  Set  off  charge  right  before  getting  on 
the  extraction  helo  (risking  the  helo  on 
deck) 

5 

Technical 

Vehicles 

Technical  vehicle  with  4  enemies 
(and  additional  troop  carrier  with  4 
additional  uniformed  personnel  if 
squad  engaged  enemies  in  Event 

3)  stops  on  road  on  the  east  side  of 
compound. 

•  Stand  their  ground  and  engage  enemy 
OPFOR 

•  Break  contact  and  move  directly  to  LZ 

4.2.3  -  Initial  Tactical  Recovery  of  Aircraft  and  Personnel  (TRAP)  Effort 

This  scenario  will  have  the  squad  performing  a  TRAP  mission  to  rescue  downed  helicopter  pilots.  They 
will  face  many  obstacles  such  as  snipers,  mortar  fire,  enemy  helicopters,  and  armored  vehicles.  This 
scenario  addresses  the  following  T&R  Objective:  INF-MAN-4209.  Table  15  displays  the  events  contained 
in  the  Initial  TRAP  Effort  scenario. 


Table  15.  Initial  TRAP  Effort  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Civilian  Truck 

Squad  begins  moving  toward  crash 
site.  A  few  minutes  later,  a  civilian 
truck  from  the  southwest  deviates 
from  the  road,  passes  the  Marines, 
and  begins  speeding  toward  the 
crash  site. 

•  Let  the  vehicle  go  and  see  what 
happens 

•  Attempt  to  disable  the  vehicle  (e.g. 
shoot  out  tires) 

•  Kill  the  driver 
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2 

Sniper  Fire 

As  the  Marines  continue  to 
approach  the  downed  helo,  a 
sniper  will  begin  firing  on  them 
while  they  are  most  likely  in  a 
relatively  open  area. 

•  Have  the  squad  take  cover  and 
communicate  to  determine  location  of 
the  sniper  before  shooting  him 

•  Immediately  suppress  the  suspected 
direction  of  the  sniper 

3 

Mortar  Fire 

After  Marines  cross  a  certain  point 
closer  to  the  helo,  they  will  begin  to 
take  mortar  fire  from  unknown  point 
of  origin  (POO),  and  will  also  hear 
gun  shots  in  the  distance. 

•  Increase  dispersion  and  move  out  of 
intended  kill  zone 

•  Take  cover  immediately  and  tuck  into 
position  until  mortar  fire  ends 

4 

Russian  Hind 

Once  the  Marine  squad  is 
approximately  200  meters  away 
from  the  downed  helo,  an  enemy 
helo  comes  up  comes  from  their 
east,  flying  towards  the  crash  site 
and  they  still  hear  bullets  nearby.  It 
appears  to  not  be  aware  of  the 
Marines'  presence. 

•  Employ  machineguns/  Shoulder- 
Launched  Multipurpose  Assault 
Weapon  (SMAW)  to  try  to  shoot 
down  the  helo  (helo  becomes 
aggressive) 

•  Remain  concealed  while  it  passes  by 

5 

Enemy  T-72 

Just  after  the  Marines  arrive  at 
downed  helo  and  begin  executing 
their  plan  to  provide  security  and 
assess  the  number  of  casualties, 
they  will  hear  and  see  an  enemy  T- 
72  roll  in  from  the  east  of  their 
location  at  about  400  south-east, 
ultimately  coming  to  a  halt  in  an 
open  area.  They  will  still  be  hearing 
bullets  flying  nearby  and  there  will 
be  dead  friendlies  around  the  helo. 

•  Stay  near  the  helo  to  provide  security 
and  prosecute  the  tank 

•  Move  to  a  position  where  the  squad 
has  more  cover  and  concealment  to 
prosecute  the  tank 

•  Attempt  to  split  the  squad  to  conduct 
both  actions  simultaneously 

4.2.4  -  Night  Ambush 

This  scenario  will  have  the  squad  conducting  an  ambush  at  night.  The  squad  will  have  many  shoot/no 
shoot  decisions,  until  the  target  finally  arrives.  This  scenario  addresses  the  following  T&R  Objectives: 
INF-MAN-4002  and  0300-PAT-1009.  Table  16  displays  the  events  contained  in  the  Night  Ambush 
scenario. 


Table  16.  Night  Ambush  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

2  Civilian 
Vehicles 

The  squad  will  have  about  5 
minutes  to  move  to  and  set  in  to 
ambush  position.  At  this  point,  2 
civilian  vehicles  drive  down  the 
Main  Supply  Route  (MSR)  in  front 
of  them. 

•  Shoot  (Marines  detected) 

•  Not  shoot 

2 

5  Civilian 
Vehicles 

About  five  minutes  after  the 
civilian  vehicles  move  through,  5- 
6  civilian  vehicles  drive  down  the 
MSR  in  front  of  the  Marines. 

•  Shoot  (Marines  detected) 

•  Not  shoot 
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3 

2  Enemy 
Vehicles 

About  five  minutes  after  the 
second  set  of  civilian  vehicles 
move  through,  two  enemy 
technical  vehicles  drive  down  the 
MSR  in  front  of  the  Marines. 

There  will  also  be  an  alarm  going 
off  in  the  distance  and  a  civilian 
runs  out  of  the  building  next  to 
the  MSR  and  runs  across  the 
road. 

•  Shoot  (Marines  detected) 

•  Not  shoot 

4 

Dismounted 

Enemies 

About  five  minutes  after  the 
technical  vehicles  move  through, 

6  dismounted  enemies  walk 
parallel  with  the  NE/SW  running 
MSR. 

•  Shoot 

•  Not  shoot  (Once  the  enemies  have 
passed  the  projected  left  lateral  limit 
(LLL)  of  the  Marines'  ambush  position, 
mortar  illumination  will  appear  above 
the  dismounts,  and  a  notional, 
adjacent  friendly  unit  will  prosecute  the 
dismounts.) 

5 

6  Enemy 

Vehicle 

Targets 

About  five  minutes  after  the 
enemy  dismounts  are  taken  out 
by  the  adjacent  friendly  unit,  6 
enemy  vehicles  come  down 

MSR. 

•  Shoot 

•  Not  shoot 

4.3  SLTP  3  -  Allah's  Purge 

A  small  Islamic-based  terrorist  group  named  Al-Sharia  has  been  impeding  the  UN's  efforts  to  provide  aid 
to  the  fictitious  country  of  Hakabaa  to  manage  an  Ebola  outbreak.  Al-Sharia  claims  the  outbreak  is  part  of 
Allah's  will  to  eradicate  the  infidels.  The  Marines  of  the  24th  Marine  Expeditionary  Unit  have  been  tasked 
with  the  neutralization  of  the  Al-Sharia  group  in  order  to  help  foster  the  quarantine  and  aid  the  UN 
medical-staff. 

4.3.1  -  Locate  and  Secure  Dr.  Platt 

This  scenario  will  challenge  the  squad  and  the  Squad  Leader  to  conduct  a  raid  and  deal  with  an 
immediate  threat  (a  sniper).  After  they  are  unable  to  secure  the  Person  of  Interest  (POI)  (Dr.  Amil  Platt), 
the  squad  takes  sniper  fire.  The  squad  will  have  to  identify  and  neutralize  the  threat.  Once  higher  directs 
them  back  to  the  starting  location,  an  enemy  force  approaches  from  the  south  to  reinforce  enemy  units 
within  the  village.  This  scenario  addresses  the  following  T&R  Objectives:  INF-MAN-4209.  Table  17 
displays  the  events  contained  in  the  Locate  and  Secure  Dr.  Platt  scenario. 

Table  1 7.  Locate  and  Secure  Dr.  Platt  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Empty  Target 
House 

The  house  is  empty.  Squad  finds 
signs  of  struggle  and  overturned 
furniture. 

•  Radio  current  sit  to  higher,  suggest 
interacting  w/  populace 

•  Radio  current  sit  to  higher 

•  Perform  extraction  w/o  radioing 
higher 

2 

Squad 

Encounters 

Boy  w /  Intel 

Squad  engages  in  conversation 
with  a  local  village  boy  that  informs 
the  Squad  Leader  that  bad  men 
took  Dr.  Platt  to  the  north  end  of 
the  city. 

•  Report  to  higher 

•  Order  the  squad  to  head  to 
compound  on  north  end  of  city 

•  Order  team  to  set  up  over-watch 
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3 

Black  SUV 
Drives  Away 

As  Marines  near  the  compound,  a 
black  SUV  drives  in  front  of  their 
view  heading  west. 

•  Request  intelligence,  surveillance, 
and  reconnaissance  (ISR)  to  follow 

SUV 

•  Open  fire  on  the  SUV 

4 

Sniper  Fire 

A  sniper  near  the  middle  of  the  city 
begins  to  engage  the  squad. 

•  Pursue  sniper  by  fire 

•  Orient  overwatch  to  sniper  position 

•  Take  cover  and  pop  smoke 

5 

Contact  from 
MGs  & 
Technical 

The  squad  is  directed  by  higher  to 
check  house  for  intel.  As  they 
approach  the  house,  they  are 
engaged  from  the  south  by  1 
technical  truck  and  4  machine 
gunners. 

•  Keep  the  squad  together  to  neutralize 
the  enemy  before  searching  for  intel 

•  Split  the  squad  before  all  enemies  are 
neutralized  to  look  for  intel 

•  Break  contact,  fall  back,  and 
discontinue  the  search  for  intel 

4.3.2  -  Clear  an  Enemy  Roadblock 

This  scenario  will  have  the  squad  attempting  to  clear  a  road  block  that  the  enemy  has  constructed  to  deny 
friendly  forces  freedom  of  movement.  The  scenario  will  escalate  as  enemy  forces  begin  firing  at  the 
squad  as  they  attempt  to  clear  the  roadblock.  After  the  roadblock  is  cleared,  a  large  enemy  force  will 
engage  the  squad’s  breaching  element  from  the  west.  This  scenario  addresses  the  following  T&R 
Objective:  INF-MAN-3202  and  INF-MAN-4204.  Table  18  displays  the  events  contained  in  the  Clear  an 
Enemy  Roadblock  scenario. 


Table  18.  Clear  an  Enemy  Roadblock  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Orange  Truck 

As  the  roadblock  comes  into  the 
squad’s  view,  an  orange  dump 
truck  drives  away  from  the  trash 
pile. 

•  Do  not  engage  the  vehicle;  observe. 

•  Report  the  truck  to  higher. 

•  Engage  the  vehicle. 

2 

Civilians  Run 
from  Trash  Pile 

The  squad  sees  several  civilians 
flee  from  the  trash  pile  as  the 
squad  nears. 

•  Observe  civilians 

•  Report  sit  to  higher 

•  Engage  civilians 

3 

Light  Fire  from 
Enemy 

As  Marines  approach  the 
roadblock,  they  receive  light 
enemy  fire. 

•  Return  fire,  suppress  enemy,  then 
send  Marines  to  set  charge 

•  Return  fire  and  simultaneously  set 
charge 

•  Move  to  set  charge  while  receiving  fire 

4 

Heavy  Fire 

During  Charge 
Placement 

An  enemy  machine  gun  team 
attacks  the  breaching  element  as 
they  are  placing  the  charge 

•  Take  cover  and  return  fire  while  talking 
the  Support  by  Fire  (SBF)  on  to  enemy 
position 

•  Have  breaching  element  return  fire  as 
main  effort 

•  Return  fire  in  an  uncoordinated  effort 

5 

Reinforcements 

After  the  roadblock  has  been 
breached,  the  breaching  element 
will  receive  heavy  fire  from  the 
west. 

•  Suppress  with  SBF  and  have 
breaching  element  withdraw 

•  Suppress  with  SBF  and  hold 
breaching  element  in  place 

•  Only  have  breaching  element  hold 
position 

4.3.3  -  Support  Third  Squad 

This  scenario  will  have  the  squad  supporting  a  pin  downed  3rd  squad.  This  will  require  the  Squad  Leader 
to  be  mindful  of  the  squad’s  effective  weapon  ranges  and  appropriately  integrate  fires.  This  scenario 
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addresses  the  following  T&R  Objective:  INF-MGUN-4001 .  Table  19  displays  the  events  contained  in  the 
Support  Third  Squad  scenario. 


Table  19.  Support  Third  Squad  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

3rd  squad 
pinned 

3rd  squad  reports  that  it  is  taking 
heavy  enemy  fire  and  needs 
immediate  support. 

•  Emplace  machine  gun  (MG)  team  to 
provide  supporting  fires  while  rest  of 
squad  maneuvers  on  to  enemy 

•  Provide  supporting  fire  with  entire 
squad  regardless  of  distance 

•  Maneuver  to  close  on  enemy  before 
providing  supportive  fires 

2 

Civilians  in 
field  of  fire 

Civilians  begin  to  run  in  a  zig-zag 
pattern  in  a  valley  below  the 
squad’s  ideal  MG  position 

•  Hold  and  shift  fires  accordingly  to 
avoid  civilians 

•  Continue  suppressing  regardless  of 
civilian  positions 

•  Engage  the  civilians 

3 

Technical 
vehicles  fire 
at  squad 

Two  technical  vehicles  adjacent  to 
the  enemy  squad  begin  descending 
down  the  hill  towards  the  squad. 

•  Request  fire  support  on  technical 
position 

•  Prioritize  technicals  with  MGs 

•  Engage  technicals  with  entire  squad 

4 

Near  threat 
fire  team 

Midway  between  the  squad  and  the 
enemy  force,  a  fire  team  begins 
making  their  way  up  to  the  squad  to 
attack. 

•  Engage  fire  team  with  light  MGs 
while  utilizing  heavy  MGs  for 
technical  or  3rd  squad  enemies 

•  Engage  fire  team  with  heavy  MGs 

5 

Enemy 
technical 
from  west 

After  all  enemies  are  neutralized,  an 
enemy  technical  attacks  the  squad 
from  the  rear. 

•  Engage  enemy  with  all  organic 
assets 

•  Call  for  danger  close  fires 

•  Break  contact  and  withdraw 

4.3.4  -  Ambush  Enemy  Reinforcements 

This  scenario  will  have  the  squad  conducting  an  ambush  on  an  enemy  force.  Environmental  conditions 
and  enemy  activity  will  force  the  Squad  Leader  to  make  tactical  ambush-preparation  decisions.  This 
scenario  addresses  the  following  T&R  Objective:  INF-MAN-4002.  Table  20  displays  the  events  contained 
in  the  Ambush  Enemy  Reinforcements  scenario. 

Table  20.  Ambush  Enemy  Reinforcements  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Fence  line 
prohibiting 
movement 

While  moving  to  the  ambush 
position,  the  squad  comes  across 
a  fence  line  with  two  breaks  in  it. 
One  of  the  breaks  is  out  in  the 
open  and  one  is  covered  by  trees. 

•  Cut  through  fence 

•  Pass  through  break  in  fence  that  is 
further  but  better  concealed 

•  Pass  through  break  in  fence  that  is 
closer  but  offers  no  cover  or 
concealment 

2 

Fog  &  rain  set 
in 

As  the  squad  settles  in  to  their 
ambush  position,  fog  and  rain  set 
in,  obscuring  their  view  of  the 
enemy  entry  point. 

•  Send  fire  team  to  an  Observation 

Post  (OP)  position 

•  Stay  in  place 

•  Move  entire  squad  to  get  better  view 
of  enemy  entry  point 

3 

Mortars  drop 

Mortars  drop  across  the  road  from 
the  squad. 

•  Stay  in  place 

•  Move  the  squad 
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4 

Ambush  target 
arrives 

The  target  enemy  force  makes  its 
way  up  the  road. 

•  Engage  the  following  troop  carriers 
first 

•  Engage  the  lead  vehicle  first 

5 

Squad  sized 
enemy  at 
extract 

After  all  enemies  are  neutralized, 
the  squad  gets  a  notice  to  extract 
from  their  insertion  point.  An 
enemy  squad  is  waiting  for  them. 

•  Set  up  a  base  of  fire  and  maneuver 
around  the  enemy 

•  Direct  all  fire  to  enemy  squad 

•  Break  contact  and  egress 

4.4  SLTP  4  -  Neo-Facist  Rising  of  Odessia 

A  Neo-Nazi  terror  organization  known  as  the  Neo-Fascist  Rising  (NFR)  has  taken  up  arms  against  the 
local  government  of  Odessia.  They  have  been  seizing  key  government  infrastructure  and  eliminating 
"sluggish"  citizens  that  they  deem  are  a  burden  to  society.  The  democratic  government  of  Odessia  has 
asked  for  outside  assistance.  Given  their  current  location  near  Istanbul  and  the  mass  killings  of 
Americans,  the  24th  Marine  Expeditionary  Unit  has  been  ordered  to  prepare  for  a  joint  operation  with  the 
Odessia  military  to  seize  back  key  terrain  and  infrastructure  from  the  NFR. 

4.4.1  -  Conduct  a  Reconnaissance  Patrol 

This  scenario  will  require  the  squad  to  conduct  a  reconnaissance  patrol  of  an  old  church  that  could  serve 
as  a  combat  outpost  for  1st  Platoon.  This  mission  will  require  the  Squad  Leader  to  recall  Commanders 
critical  information  requirements  (CCIRs)  to  aid  in  decision-making  when  faced  with  unexpected  events. 
This  scenario  addresses  the  following  T&R  Objectives:  0300-PAT-1009  and  INF-MAN-4301.  Table  21 
displays  the  events  contained  in  the  Conduct  a  Reconnaissance  Patrol  scenario. 

Table  21.  Conduct  a  Reconnaissance  Patrol  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Distant 

explosions 

Explosions  are  heard  in  the 
distance  at  the  beginning  of  the 
recon  patrol 

•  Continue  pushing,  update  higher 

•  Ignore  explosions,  continue  pushing 

•  Explore  the  explosion  source 

2 

Celebratory 

gunfire 

Erratic  gunfire  is  heard  in  the 
distance 

•  Continue  pushing,  update  higher 

•  Send  a  fire  team  to  investigate 

•  Move  entire  squad  to  investigate 

3 

Technical  at 
road  crossing 

A  technical  vehicle  comes  down  a 
road  the  squad  is  about  to  cross 

•  Hide,  do  not  engage 

•  Engage  vehicle 

4 

Technicals 
open  fire  on 
squad 

After  crossing  the  road,  multiple 
technicals  open  fire  on  the  squad 
from  the  road 

•  Suppress  and  break  contact  to 
withdraw  to  objective 

•  Dig  in,  return  fire,  and  neutralize 
vehicles 

•  Call  for  support  from  an  adjacent  unit 

5 

Enemy 

fighters 

engage 

squad 

The  squad  takes  gunfire  from 
enemies  after  securing  the 
compound 

•  Suppress  and  envelop  to  secure  the 
compound 

•  Engage  the  enemy  from  one  location 
to  secure  the  compound 

•  Suppress  and  break  contact  to  move 
towards  extraction 

4.4.2  -  Secure  a  Bridge 

This  scenario  will  have  the  squad  attempting  to  secure  a  bridge  to  prevent  enemy  movements.  The 
Squad  Leader  will  have  to  make  various  decisions  in  how  to  respond  to  enemy  activity  and  how  to 
prioritize  enemy  combatants.  This  scenario  addresses  the  following  T&R  Objective:  INF-MAN-4302.  Table 
22  displays  the  events  contained  in  Secure  a  Bridge  scenario. 
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Table  22.  Secure  a  Bridge  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Civilians 

fleeing 

3-4  civilians  are  seen  running 
towards  the  squad  from  about  200- 
300  meters  away. 

•  Question  civilians 

•  Do  not  engage 

•  Engage 

2 

Civilian 

execution 

An  enemy  lines  up  a  few  civilians 
with  a  gun  pointed  towards  them. 

•  Detain  enemies 

•  Squad  to  shoot  enemy 

•  Disregard,  continue  on  mission 

3 

NFR  Boat 
Sighting 

An  enemy  boat  comes  up  the  river 
towards  the  bridge  and  turns 
around  to  go  back  upstream. 

•  Observe  and  report  enemy  activity 

•  Ignore  boat 

•  Engage  the  boat 

4 

MG  fire  &  fire 
team  while 
squad  on  the 
bridge 

An  enemy  machine  gunner  &  fire 
team  opens  fire  on  the  squad  as 
they  near  the  bridge. 

•  Suppress  and  use  precision  fire  to 
eliminate  the  threat 

•  Obscure  with  smoke 

•  Bound  to  break  contact 

5 

5  Ton  and 

BTR 

(armored 

personnel 

carrier) 

Once  the  machine  gunner  is  dead, 
the  squad  encounters  a  5  ton  full  of 
troops  and  a  BTR  approaching  the 
bridge 

•  Suppress  enemy  infantry  &  destroy 
armor  with  anti-armor  weapons 

•  Ignore  infantry  until  armor  is 
neutralized 

•  Prioritize  enemy  infantry  as  target 

4.4.3  -  Destroy  a  Weapons  Cache 

This  scenario  will  have  the  squad  searching  a  house  for  an  enemy  weapons  cache.  The  Squad  Leader 
will  have  to  determine  how  to  proceed  when  the  weapons  cannot  be  found  and  enemy  contact  is  made. 
This  scenario  addresses  the  following  T&R  Objective:  INF-MGUN-4213.  Table  23  displays  the  events 
contained  in  the  Destroy  a  Weapons  Cache  scenario. 


Table  23.  Destroy  a  Weapons  Cache  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Scooter 
leaves  dry 
hole 

A  scooter  leaves  a  house  fitting 
target  description 

•  Let  the  vehicle  go  and  observe  what 
happens 

•  Kill  the  driver 

2 

Dry  hole 

The  closest  house  fitting  the  target 
description  is  empty 

•  Expand  search  to  buildings  with 
similar  descriptions 

•  Search  all  nearby  buildings 

•  Call  higher 

3 

Two  men 
fleeing  house 

Two  unarmed  men  in  enemy  garb 
flee  another  house  fitting  the  target 
description 

•  Do  not  shoot  the  men 

•  Shoot  the  men 

4 

Booby  trap  in 
house 

Barrels  of  explosives;  notification 
of  bomb  in  the  house 

•  Get  full  cover  behind  an  obstacle  (i.e. 
log  pile,  barn,  truck) 

•  Run  away  from  house  without 
seeking  cover 

•  Do  nothing  or  attempt  to  diffuse  bomb 

5 

Mortar  & 

ambush 

attack 

Mortar  rounds  and  heavy  machine 
gun  fire 

•  Regroup,  set  up  defense  outside  of 
mortar  kill  zone.  Use  base  of  fire  and 
maneuver  element  against  threat. 

•  Regroup,  order  squad  to  pull  back  to 
a  hasty  defense  and  engage  targets 
as  they  appear. 

•  Set  up  hasty  defense  in  the  kill  zone. 
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4.4.4  -  Conduct  a  Ground  Attack 

The  squad  is  charged  with  operating  as  part  of  a  Battalion  ground  attack.  The  squad’s  mission  is  to 
overtake  an  observation  post  without  alerting  an  enemy  and  support  Alpha  Company’s  assault  on  an 
objective.  Though  the  squad  is  not  forewarned,  if  the  squad  opens  fire  on  Alpha  Company’s  objective 
before  Alpha  Company  begins  their  assault,  the  squad  will  get  into  a  battle  in  which  the  odds  are  not  in 
their  favor.  This  scenario  addresses  the  following  T&R  Objectives:  031 1  -OFF-2002  and  INF-MAN-4001. 
Table  24  displays  the  events  contained  in  the  Conduct  a  Ground  Attack  scenario. 

Table  24.  Conduct  a  Ground  Attack  Events 


Event 

Event  Title 

Stimuli 

Decision  Alternatives 

1 

Vehicle 

breakdown 

The  squad’s  vehicles  break  down 
in  route  to  the  dismount  location 

•  Begin  movement  on  foot  &  then 
report  to  higher 

•  Troubleshoot  vehicle 

•  Call  higher  for  guidance 

2 

Secure  the 

OP 

The  squad  observes  fire,  smoke, 
and  dogs  as  they  move  toward  the 
OP 

•  Skirt  rocks  downwind  to  capture 
camp 

•  Approach  OP  from  any  other  direction 
than  SW 

3 

Mortar 
positions  at 
enemy 
objective 

The  enemies  have  mortars  pointed 
in  the  direction  of  Alpha  Company 

•  Call  for  fire  on  mortar  positions 

•  Directly  engage  mortar  positions 

•  Ignore  mortar  positions 

4 

Friendly  Fire 
from  Light 
Armored 
Vehicles 
(LAVs) 

The  squad  receives  friendly  fire 
from  Alpha  Company  LAVs 

•  Radio  to  higher  &  request  cease  fire 

•  Take  cover  and  wait  it  out 

•  Engage  the  LAVs 

5 

Enemy 

Retreats  from 
Objective  1 

The  enemy  begins  to  retreat  south 
away  from  Objective  1 

•  Hold  in  the  OP  &  deny  enemy  egress 
to  the  south 

•  Hold  in  place  &  engage  targets  of 
opportunity 

•  Assault  objective  1  on  foot 

5.0  STAR-DM  Assessment  and  Debrief  Tool 

Assessment  of  DM  skills  in  real-time  during  an  exercise  can  be  quite  challenging  for  a  few  reasons, 
including  that  1)  decisions  are  sometimes  made  very  quickly  and  are  therefore  easy  to  miss,  and  2)  much 
of  the  reasoning  behind  decisions  happens  in  the  mind  of  the  decision-maker  and  is  often  forgotten  by  the 
time  the  scenario  is  over,  meaning  that  assessment  can  generally  only  be  based  on  the  action  taken  and 
outcome  of  the  decision.  This  also  makes  it  more  difficult  to  provide  meaningful  process-level  feedback, 
as  the  first  3  steps  in  the  OODA  model  of  decision-making  are  lost.  The  use  of  turnkey  simulation-based 
scenarios  with  specific  decision  events  provides  the  opportunity  to  clearly  lay  out  for  the  instructor  (or 
whoever  is  conducting  the  assessment)  what  the  decision  events  are  and  what  aspects  of  the  scenario 
may  contribute  to  the  OODA  process.  This  provides  a  reference  point  from  which  to  conduct  the 
assessment  and  facilitate  the  feedback  during  the  AAR.  In  the  SLTPs,  this  opportunity  led  to  the 
development  of  the  Assessment  and  Debrief  Tool,  a  tool  to  make  the  DM  and  learning  strategies  more 
accessible  and  easy  to  implement.  The  tool  has  evolved  from  a  paper-based  checklist  into  a  mobile 
tablet-based  software  program  that  interfaces  with  VBS2  to  pull  additional  performance  metrics. 

5.1  Paper-based  Checklist 

The  checklist  is  arranged  in  such  a  way  that  the  instructor  can  easily  collect  and  scan  information  about 
the  Squad  Leader's  DM  process,  and  use  the  form  to  facilitate  a  process-level  debrief  regarding  the 
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strengths  and  weaknesses  of  the  Squad  Leader's  decision-making.  The  paper-based  assessment 
checklist  was  used  in  the  field  study  by  Marine  instructors  and  refined  based  on  their  feedback.  The 
strong  upside  to  this  version  is  that  it  requires  no  extra  technology  to  implement.  The  downside  is  that  the 
instructor  is  either  tasked  with  or  unable  to  keep  track  of  minute  simulation  details  such  as  the  event 
number  or  the  number  of  times  the  squad  is  hit  by  enemy  fire. 

Figure  32  provides  an  example  of  the  checklist  for  one  decision  event  in  Scenario  2  of  SLTP  1.  The 
checklist  for  each  decision  event  includes  five  rows  that  correspond  to  the  Observe,  Orient,  Decide,  Act, 
and  Outcome  aspects  of  the  decision.  As  the  instructor  observes  the  Squad  Leader  communicating  with 
the  squad  during  the  scenario,  he  can  listen  for  each  of  the  items  in  the  checklist  to  determine  if  the 
Squad  Leader  detects  the  important  cues  in  the  environment  (Observe),  recognizes  the  implications  of 
the  cues  (Orient),  considers  various  courses  of  action  (Decide),  and  takes  the  appropriate  action  (Act). 
The  instructor  can  check  off  each  of  these  during  the  decision  event,  and  then  observe  the  outcome  of 
the  action  that  is  taken  and  record  that  as  well.  The  options  listed  in  the  checklist  were  developed  by 
working  with  instructors  from  the  ISULC  to  try  to  anticipate  the  most  relevant  options  that  may  occur  for 
Squad  Leaders  with  a  variety  of  experience.  Therefore,  some  options  are  better  than  others.  There  is  also 
an  “Other”  option  in  case  something  else  comes  up  during  a  scenario  that  is  not  covered  in  the  checklist. 
Additionally,  there  is  a  “Notes”  section  for  the  instructor  to  quickly  jot  down  thoughts  he  would  like  to 
revisit  for  that  decision  event  during  the  AAR. 


Participant  Number 


SLTP  1  Scenario  2:  Platoon  Ambush 

Check  V  appropriate  boxes 

Event  1:  Civilians  Passing 

Detects 

O  Individuals 

CD  Other: 

Recognizes 

D  People  are 
civilians 

CH  Enemy  will  be  in 
uniform 

O  Enemy  will  be 
platoon  size 

□  Other: 

Courses  of 

Action 

Do  not  shoot 

O  Considered 

D  Chosen 

Engage  w/squad  weapon 
O  Considered 

D Chosen 

Engage  with  IDF 

O  Considered 

O  Chosen 

Other: 

O  Considered 

0  Chosen 

Actions 

Taken 

Order  squad  to  not 
shoot 

D  Performs 

Order  squad  to  engage 
with  squad  weapons 

O  Performs 

Orders  squad  to 
engage  with  IDF 

O  Performs 

Other: 

D  Performs 

Outcome 

Civilians  are  safe 

O  Occurs 

Civilians  are  harmed 

O  Occurs 

Squad  is  undetected 
□  Occurs 

Other: 

O  Occurs 

Notes: 

Figure  32.  Example  of  Decision  Event  Portion  of  the  Paper-based  Checklist 


In  addition  to  the  checklist  for  each  decision  event  in  a  scenario,  the  paper-based  checklist  ends  with 
some  metrics  for  the  Squad  Leader’s  performance  in  the  overall  mission  (scenario).  These  items  measure 
observation  skills,  situational  assessment,  adaptability,  and  effectiveness.  Each  item  in  this  portion  is 
rated  essentially  from  1-10  according  to  the  categories  shown  in  Figure  33  (Unsatisfactory-Outstanding). 
The  final  portion  of  the  paper-based  checklist,  shown  in  Figure  34,  allows  the  instructor  to  rate  the 
difficulty  of  the  mission  to  potentially  provide  some  perspective  on  his  assessment  of  the  Squad  Leader 
(e.g.,  if  the  Squad  Leader  did  poorly  but  it  was  a  difficult  mission,  the  poor  performance  may  not  be  as 
concerning  as  poor  performance  on  an  easy  mission).  The  final  portion  also  provides  a  place  for  the 
instructor  to  jot  down  any  additional  notes  about  the  overall  mission  for  him  to  revisit  during  the  AAR. 
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Mission  Outcome 

Observation  Skills  -  Overall  ability  to  report  the  cues  needed  to  formulate  a  decision: 


Unsatisfactory 

Below  Average 

Average 

Above  Average 

Outstanding 

1 

1 

1 

Situational  Assessment  -(METT-T  analysis,  EMLCOA,  SOM  FSP,Task  &ME,  Commanders  Intent) 


Unsatisfactory 

Below  Average 

Average 

Above  Average 

Outstanding 

Adaptive  Decision  Making-  Ability  to  adapt  to  changes  in  terms  of  enemy  action,  friendly  situation,  and  environment. 


Unsatisfactory 

Below  Average 

Average 

Above  Average 

Outstanding 

1 

1 

1 

Effectiveness  -  Overall  all  ability  to  manage  and  employ  subunits  and  resources  to  achieve  commander's  intent 


Unsatisfactory 

Below  Average 

Average 

Above  Average 

Outstanding 

1 

1 

1 

Figure  33.  Mission  Outcome  Metrics 


Mission  Difficulty 

Mission  Difficulty  -  Overall  difficulty  of  the  mission 


Very  Easy 

Easy 

Average 

Difficult 

Very  Difficult 

1 

1 

1 

Notes: 


Figure  34.  Mission  Difficulty  Rating  and  Mission  Notes 


5.2  Software-based  Assessment  and  Debrief  Tool 

The  second  version  of  the  assessment  checklist  is  a  Windows-based  software  tool  that  can  be  run  on 
either  a  PC  or  a  Windows  tablet.  It  contains  the  decision  event  checklists  for  every  scenario  in  SLTPs  1  -4, 
is  connected  with  the  VBS2  simulation  to  monitor  simulation  details  such  as  the  event  number  and  a 
variety  of  performance-based  metrics,  and  has  a  checklist  editor  so  that  the  instructor  can  make  edits  or 
create  new  checklist  assessments.  The  software  version  of  the  Assessment  and  Debrief  Tool  (ADT)  is 
designed  to  allow  the  instructor  to  quickly  assess  the  Squad  Leader's  DM  process  and  reference 
simulation  metrics  when  appropriate  to  facilitate  the  Squad  Leader's  training.  It  has  the  added  benefits 
over  the  paper-based  checklist  of  1)  being  able  to  rate  the  competence  of  the  decisions  at  the  process- 
level  for  each  event  in  addition  to  checking  off  items  on  the  checklist,  2)  allowing  the  instructor  to  review 
the  VBS2  metrics  in  real-time  to  help  provide  context  during  sometimes  rapidly  unfolding  scenarios,  and 
3)  providing  an  automatic  AAR  screen  for  the  instructor  that  quickly  summarizes  his  ratings  to  facilitate  an 
immediate  and  thorough  debrief.  The  software  version  does,  however,  require  an  additional  computer  or 
tablet. 

As  shown  in  Figure  35  below,  the  home  screen  for  STAR-DM  ADT  is  where  you  can  access  all  of  the 
different  options  the  program  offers.  There  are  five  different  pages  to  choose  from:  Assessments,  Load 
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AAR  (After  Action  Review),  Assessment  Builder,  Help,  and  Log  Out.  Assessments  monitors  live  scenarios 
and  has  input  for  evaluation  tracking.  Load  AAR  allows  the  instructor  to  reassess  existing  AARs  at  any 
time.  Assessment  Builder  allows  you  to  create  a  new  scenario.  Help  provides  a  document  with  essential 
information.  Log  Out  returns  you  to  the  Log  In  screen. 


Figure  35.  ADT  Home  Screen 


Assessments 

Assessments  in  STAR-DM  ADT  allow  the  instructor  to  follow  and  review  a  student’s  performance  in  real 
time.  These  assessments  allow  the  instructor  to  view  live  scenario  events,  to  take  notes  (typed  and  hand 
written),  and  to  rate  the  student’s  task  execution.  The  Assessment  page  (Figure  36)  is  separated  into 
three  main  sections  -  Current  Scenario,  Scenario  Events,  and  Events.  The  current  scenario  is  what 
determines  the  content  that  will  be  shown  for  the  events.  This  is  the  first  choice  to  make  when  arriving  to 
the  Assessments  page.  The  scenario  will  begin  once  the  student  has  started  their  session  on  the  Trainee 
Squad  Leader  Computer.  In  the  Scenario  Events  section,  a  “Scenario  Start”  timestamp  will  appear  once 
the  scenario  has  begun.  The  Scenario  Events  section  (top  of  the  screen  in  Figure  36)  holds  the  real-time 
information.  Events  will  be  written  out  per  line,  with  a  timestamp  and  information  about  what  happened  at 
that  time.  These  are  helpful  for  reviewing  exact  times  for  events  for  ratings.  The  timestamps  will  indicate 
when  a  scenario  and  event  has  begun  or  ended.  Events  in  a  scenario  are  categorized  according  to  time 
and  action.  Each  scenario  has  five  events.  To  change  an  event,  the  user  selects  the  event  button  and  the 
corresponding  table  will  appear.  An  Event  table  has  four  columns:  steps,  actions  taken,  notes,  and  overall 
rating.  Within  actions,  the  checklists  allow  the  instructor  to  correspond  student  activities  with  completion. 
Notes  are  helpful  for  leaving  extra  information.  Note  buttons  are  available  for  each  Event  and  also  have 
the  ability  to  leave  handwritten  notes.  Overall  ratings  for  each  step  can  be  assigned  with  four  different 
ratings:  Good,  OK,  Poor,  or  Not  Sure. 


67 


□ESIGNinteractive 


Design  Interactive,  Inc. 


Contract#:  N00014-12-G-0427 


Figure  36.  Assessment  Screen 


After  Action  Review 

AARs  are  a  debrief  process  for  analyzing  what  happened  and  why  it  happened.  STAR-DM  ADT’s  AAR 
provides  the  instructor  with  a  summary  and  details  of  trainee  performance  for  each  scenario.  Instructors 
can  view  their  notes,  ratings,  and  statistics  gathered  from  the  scenario.  The  AAR  also  allows  saving  and 
loading,  a  useful  tool  to  review  summaries  when  needed.  On  the  home  page,  a  Load  AAR  button  allows 
the  instructor  to  reassess  existing  AARs  at  any  time.  Within  an  Assessment,  the  AAR  button  within  the 
Assessment  menu  (located  in  the  top  blue  bar)  will  automatically  open  the  AAR  related  to  the  current 
Assessment.  The  AAR  is  laid  out  in  three  different  sections:  Scenario  Summary,  Event  Results,  and 
Assessment  Notes,  see  Figure  37.  The  scenario  summary  shows  the  comprehensive  numbers  for  the 
entire  scenario.  The  metrics  are  listed  downwards  on  the  left,  and  the  corresponding  results  are  on  the 
right.  The  event  results  is  comprised  of  two  separate  sections  -  Checklist  Results  and  Metric  Results. 
Both  of  these  sections  review  the  individual  outcomes  and  metrics  for  each  event.  The  checklist  results 
review  the  instructor-determined  overall  ratings  for  the  steps  within  each  event.  The  green  checkmark  is 
Good,  the  yellow  warning  is  OK,  and  lastly,  the  red  X  is  Poor.  The  metrics  results  for  the  event  results  are 
similar  to  the  scenario  summary,  but  are  detailed  for  each  individual  event.  This  elaborates  on  the 
scenario  summary  and  provides  further  detail.  The  assessment  notes  section  provides  a  scrollable 
window  to  review  instructor-taken  notes  from  the  assessment.  The  notes  are  structured  linearly,  from  the 
first  event  to  the  last.  Also,  the  instructor  can  load  written  notes.  Saving  an  AAR  enables  the  AAR  to 
become  available  for  review  in  the  future. 
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SLTP  2  Scenario  1:  Clearing  Mission 


Scenjro  Summiiy 


Mm 

MosonTiire 

OOCOOD 

Pfeitcpar*  Slicti  fired 

ti 

Pert*  per*  Shot  Ophor 

16 

Pare*  par*  Ac&reev 

1*28% 

Opfor  Snots  Find 

142 

Opfor  Snot  Participant 

2 

Opfcr  doc 

19 

®eit«'par«  Shot  Civilian 

0 

Crvilan  Cited 

3 

Assessment  Scans 

Scenaro  Note; 

The  student  in  thsKCnird 

dill  Mil 

Aeons  Iran 


Courses  o'  Act©* 


[Meets 

Recognises: 

Aeons  Tekerv 


Jlnemt^lCrriienOsrtsmto  ctvrtue  Shooang  Ipwlaig  Truck  Wpetton  the  Ml  Emmy  Tanks  and  Techascal 

dicisicw 

© 

© 

A 

© 

Observe 

© 

© 

© 

© 

© 

- 

A 

© 

Q 

© 

A 

Decide 

© 

© 

A 

© 

Ad 

© 

A 

G 

© 

© 

Mlll^  ?Fnemy.ndlCMIImiOi*tJ*e  Ci.dan  Oi—I-hi  SpnPRrq  ,,**  Smpm  o*  tfv  »M  Enemy  Teido  and  tadmol 

003042 

0301:10 

000*16 

03004S 

Participant  Shots  Feed 

■ 

s 

6 

32 

39 

Aarticipent  Shot  Cpfej' 

0 

2 

’ 

9 

« 

Partcipant  decency 

0% 

«% 

1667% 

2412% 

1026% 

Oofer  Shots  fired 

0 

0 

0 

25 

317 

Cp"or  Slso*  Faroe  cert 

• 

0 

• 

| 

- 

Opto,  reed 

0 

1 

- 

6 

" 

Participant  Shot  Cvhan 

• 

0 

• 

0 

0 

Cwslan  Cited 

" 

• 

■ 

■ 

" 

Figure  37.  AAR  Screen 


Assessment  Builder 

The  Assessment  Builder  (Figure  38)  gives  the  instructor  an  opportunity  to  author  scenario  assessments.  It 
allows  the  instructor  to  input  details  for  five  events.  The  only  constraints  applied  to  the  scenario  builder 
are  those  that  are  based  on  the  SLTP  training  framework.  1)  Each  scenario  must  contain  five  events.  The 
events  are  listed  in  chronological  order  and  are  helpful  for  creating  student  action  checklists.  2)  All  events 
must  have  at  least  one  item  for  each  category:  Detects,  Recognizes,  Courses  of  Action,  Actions  Taken, 
and  Outcome. 


Charlie  Peter 

Scenario  Name 

Event  Setup 

Event  1  Event  2  Event  3  Event  4  Event  5 

Event  Name: 

Detects: 

Recognizes:  ^^^B 

Courses  of  Action:  B  ^^^B 

Actions  Tbken:  B  ^^^B 

Outcome:  ^^^B 


Saw  Chocklist 

Figure  38.  Assessment  Builder 
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6.0  Transition  and  Way  Forward 

A  formal  Technology  Transition  Agreement  (TTA)  has  been  created  for  STAR-DM  and  iterated  with 
transition  partners.  It  has  not  yet,  however,  been  approved.  One  potential  issue  that  has  come  up  is  that 
the  SLTPs  were  developed  in  VBS  version  2.15  and  subsequent  to  their  development,  the  Marines  have 
upgraded  many  of  their  computers  to  VBS3.  There  are  some  backwards-compatibility  issues  with  VBS3 
which  would  require  updates  to  the  SLTPs  in  order  for  them  to  fully  function  in  the  new  version. 
Therefore,  it  is  recommended  that  the  SLTPs  be  upgraded  to  the  current  VBS  version  in  order  for  them  to 
be  readily  used.  However,  the  SLTPs  have  still  been  transitioned  to  a  few  locations.  Both  Camp  Upshur 
and  the  IIMEF  Simulation  Center  were  involved  in  a  SLTP  evaluation  event  and  received  copies  of  the 
SLTPs  and  all  supporting  material.  Additionally,  the  Combating  Terrorism  Technical  Support  Office  and 
the  Camp  Pendleton  NT  have  received  copies  of  the  SLTP  training  packages  as  well. 

In  terms  of  future  research,  there  is  still  a  desire  to  conduct  the  planned  TEE  to  fully  evaluate  the  training 
effectiveness  of  the  complete  STAR-DM  training  framework  as  well  as  to  determine  the  return  on  training 
investment  for  the  multiple  training  environments  of  interest.  It  is  recommended  that  the  program 
sponsors  continue  to  pursue  available  participants  for  the  execution  of  such  a  study  in  a  follow-on  effort. 

Furthermore,  the  addition  of  a  simple  socio-evaluative  stressor  prior  to  simulation-based  performance  led 
to  significant  increases  in  physiological  stress  response.  Stress  response  was  effectively  captured  via 
electrodermal  and  cardiovascular  measures  of  heart  rate  and  skin  conductance  level.  Further,  an 
algorithm  which  assesses  changes  in  heart  rate  and  skin  conductance  level  to  quantify/qualify  an 
individual’s  short  term  resilience  to  stress  was  developed  and  revealed  the  ratio  of  baseline  perceived 
stress  to  baseline  cortisol  levels  are  potentially  effective  predictors  of  an  individual’s  resilience  to  stress. 
Further  research  is  needed  to  explore  the  effectiveness  of  such  methods  at  predicting  resilience. 
Continuing  work  is  focused  on  adapting  this  training  to  military  personnel,  and  assessing  the  utility  of 
various  coping  and  decision-making  strategies  on  performance  and  physiological  stress. 
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Appendix  A.  Key  Terms 


AAR 

After  Action  Review 

AITB 

Advanced  Infantry  Training  Battalion 

AO 

Area  of  Operation 

AOI 

Area  of  Interest 

BARS 

Behaviorally  Anchored  Rating  Scale 

BFT 

Bio  Feedback  Training 

CASE 

Collect  data,  Assess  situation,  Select  response,  Evaluate  response 

CD-RISC 

Connor-Davidson  Resilience  Scale 

COA 

Course  of  Action 

COC 

Command  Operations  Center 

CTTSO 

Combating  Terrorism  Technical  Support  Office 

DM 

Decision  Making 

DMAT 

Decision  Making  Assessment  Tool 

ECG 

Electrocardiogram 

EDA 

Electrodermal  activity 

EDR 

Electrodermal  Response 

EMG 

Electromyogram 

FINEX 

Finish  of  Exercise 

FRAGOs 

Fragmentary  Orders 

HRV 

Heart  Rate  Variability 

NT 

Infantry  Immersion  Trainer 

IPG 

Instructor  Preparation  Guide 

IRR 

Interrater  Reliability 

ISULC 

Infantry  Small  Unit  Leader  Course 

MAGTF 

Marine  Air-Ground  Task  Force 

METT-TC 

Mission,  Enemy,  Terrain  &  Weather,  Troops,  Time  Available,  and  Civilian  Considerations 

MSEL 

Master  Scenario  Event  List 

00  DA 

Observe,  Orient,  Decide,  Act 

PPG 

Pulse  Plethysmography 

PTT 

Pulse  Transit  Time 

ROTI 

Return  on  Training  Investment 

RSA 

Respiratory  Sinus  Arrhythmia 

S&T 

Science  and  Technology 

SECPT 

Socially-Evaluated  Cold  Pressor  Test 

SET 

Stress  Exposure  Training 

SHOR 

Stimulus,  Hypothesis,  Options,  Response 

SJT 

Situational  Judgment  Tests 

SLTP 

Squad  Leader  Training  Package 

SME 

Subject  Matter  Expert 

SOI-E 

School  of  Infantry  -  East 

STAR-DM 

Small-unit  Training  for  Adaptability  and  Resilience  in  Decision  Making 

STEX 

Sand  Table  Exercise 

STO 

Science  and  Technology  Objective 

STOPS 

Strategic  Operations 

T&E 

Training  and  Education 

T&R 

Training  and  Readiness 

TDG 

Tactical  Decision  Game 

TEE 

Training  Effectiveness  Evaluation 

TSST 

Trier  Social  Stress  Test 

TTPs 

Tactics,  Techniques  and  Procedures 

USMC 

United  States  Marine  Corps 
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Appendix  B.  Decisions  Library 


Situation 

Task/ 

Mission 

Objective 

Decision 

Prerequisite 

Knowledge 

Affecting 

Data  to  Collect  / 
Cues  to  Observe 
(METT-T 

Analysis) 

Orient/ 

Situation 

Assessment 
(what  does  this 
mean) 

Decide/ 

Potential  COAs 
(Pros/Cons) 
(Decision 
Alternatives) 

Which 

Action  did 
you  take? 
Why? 

Outcome(s) 

What  factors 

made  it 

stressful 

(environments 

1  factors  vs. 
task  factors) 

Marine  in  squad  lost  control 
of  a  weapon  that  was  firing 
wildly;  weapons  malfunction 

Decision  was 

made  to 
physically 

remove 

Marine  from 

behind  the 

weapon  so 

others  could 
gain  control 

Ldr  observed  gun 
barrel  pointed  at 
Marines;  observed 
Marine  at  gun  in 
awkward  body 
position;  observed 
Marine  at  gun 
white  knuckling  the 
weapon  unable  to 
control  it 

Realized  the 
weapon  was  out 
of  control  and  all 

of  the  Marines  in 
the  vicinity  were 
in  danger 

Get  shot;  Hit  the 
deck  without 

concern  for 

others 

Physically 
removed 
the  Marine 

from  behind 
the  gun 

Safely  took 
control  of  the 

weapon 
without  injuring 
other  Marines 
except  the  one 
that  needed  to 
be  physically 
removed 

emotional 

connection  to 
squad;  large 
barrel  pointed 
at  you  from 
close  range; 
unanticipated 
situation 

Former  Marine  working  as 
Contractor  providing 
Executive  protection  in 

Mexico  for  CEO's,  VIPs,  etc.. 

Transport 

HVI 

between 

locations 

Which  route 

to  take 

Knowledge  of  the 
danger  of  the  area; 
tactics  used  by 
Cartels;  ability  to 
use  children  as 
intel  agents 

Built  positive 
relationships 
with  children  in 
the  area;  Child 
ran  up  to  convoy 
and  alerted  them 

to  violence  that 

was  about  to 
take  place  on  a 
particular  route. 

Contractor  - 
trusting  child 
realized  this 
could  put  the 
convoy  and  HVI 
in  jeopardy 

ACOAs 

include 
proceed  on 
planned 
route;  take 
detour  to 

avoid 

possible 

violence. 

Chose  to  take 

detour  to 

ensure  the 
safety  of  the 
HVI. 

Convoy 

continued 

without 

incident.  There 
was  escalating 
violence  on  the 
original  route 
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Doing  training  op  with  Thai, 
so  gun  trucks  and  LAR,  in 
support  by  fire/  overwatch, 
got  dusty/cloudy  and  no  one 
could  see  where  friendlies 
were 


Fire  on  Should  he 
objective  fire? 


Deconflict 

other 

elements 

attached 

and  trying 

to 

maneuver 
on  an 
objective 


friendlies 

would  be 
maneuvering 
on  the 
objective 

dusty/cloudy  from 
LARs 

he  could  not  see 
the  objective 
well  and  couldn't 

see  friendlies  - 
so  he  might  hit 
them 

Shoot  without 
visibility, 
coordinate  to  try 
to  increase 
visibility  -  he 
talked  to  LAR 

and  asked  them 

to  slow  rate  of 
fire  to  get  eyes 
on  objective;  got 

maneuver 

elements  on 

comm  to  find  out 
if  they  were  in 
place 

increased 
visibility  on 
objective  and 
knowledge  of 
location  of 
friendlies 

dusty/cloudy 
with  friendlies 
coming  in; 
stressful 
knowing 
another 
countries'  guys 
were  attached  - 

not  sure  what 
they  would  do 

knows 

where 

friendlies  are 

located 

objective 

objective  is  clear 
-  so  he  can  fire 
weapon  without 
endangering 
friendlies 

Fired  the  missile; 
could  have 

chosen  different 
weapon  system 
if  friendlies  were 
within  danger 
area;  could  have 
cut  off  fires 

Firing 

missile  was 

best 

decision  - 

was 

mission, 
and  correct 

measures 

had  been 

taken  to 

deconflict 

mission  was 

successful 
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Engaging  targets  outside  of 
max  effective  range  of 

Javelin  system,  did  not  have 
clear  field  of  view  -  knew  that 

there  were  enemies  but  also 
children,  being  actively 
engaged  at  the  time;  did  not 
know  if  officer  knew  that 

there  were  civilians 

children  may 
be  in  vicinity 
of  target 

Shoot  or  just 
wait  until 
weapon  is 
ineffective 

not  clear  field  of 
view,  knew  that 
there  were 
children;  did  not 
know  if  higher  who 
gave  the  go-ahead 
to  shoot  knew  that 

there  were 
children;  snipers 
next  to  him  that 

could  take  out  the 

enemies 

might  hit  children 

Waited  for 
weapon  to 
expire;  could 
have  shot 

enemies  were 
taken  out  by 
other  means 

Time  pressure  - 
4  min  for 
weapon  to  be 
fired  before 
ineffective; 
possible 
women  and 

children  that 
could  be  hit; 
limited  visibility 

While  a  team  leader,  on  a 
hilltop  looking  over  a  small 
village,  like  guardian  angels 
for  dismounted  and  mounted 
patrols,  watched  area  for  6-8 
hours  a  day  for  a  few  days 
while  patrols  would  travel 
through.  On  3rd  day,  at  an 
intersection  saw  a  farmer 
start  digging  next  to 
intersection  a  couple  of 
hours  before  patrol  would 
travel  through.  Farmer 
started  emplacing  in  ground. 
Superior  told  not  to  engage 
the  farmer. 

Killed  farmer 
despite 
being  told 
not  to  by 
superior 

farmer  at 

intersection 
digging  a  hole  and 
putting  something 
in  it 

farmer  probably 
placing  IED  and 
would  probably 
place  other  lEDs 
in  the  future 

decided  to  shoot 

farmer  was 

killed 

Higher  would 
not  allow  to  fire 
even  though 
everyone  on 

the  team  saw 
the  IED 
emplaced 
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Mobile  assault  platoon  - 
truck  got  blown  up  by  lEDs 
and  lost  comms  and  needed 

to  comm  with  trucks  behind 

communic 

ate  with 

trucks 

through 

hand 

signals 

Outside  a  patrol  base  and 
one  squad's  comm  went 
down  and  then  went  back  to 
base,  they  walked  into  an 
ambush,  hit  on  2  sides 

what  to  do 
when  other 
squad  being 
shot  and 
superiors 
trying  to  tell 
people  what 
to  do  when 
he  was  patrol 
leader 

heard  shots 

other  squad 
taking  fire 

had  LT  call  in 

indirect  fire  to 

take  out 

enemies 
shooting  at 
squad 

squad  got  out 
ok 

time  pressure; 
friendlies  under 
fire;  superiors 
on  patrol  with 
squad  were 
trying  to  tell 
people  to  do 
things  that  did 
not  mesh  with 
Squad  Leader's 
plan;  was  first 
time  in  combat 
for  one  superior 
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truck  behind  him  got  blown 
up  -  gunner  saw  a  guy 
standing 

whether  to  tell 
the  gunner  to 
shoot 

usually  if  first 
truck  in  a 
convoy  does 
not  blow  up, 
it  means  that 

there  is  a 
trigger  man; 
in  that  AO, 
trigger  man 
usually  runs 
away  after 
setting  off 
trigger 

man  standing  near 
where  the  IED 

went  off 

man  is  probably 
not  the  trigger 
man  for  the  IED 

detained  the 

man  instead  of 
shooting  him 

when  man  was 
detained,  turns 
out  he  was  not 
the  triggerman 
(it  was  a 
pressure  plate 
IED),  but  he 
was  Taliban 

Battalion  set  up  in  blocking 
position  for  CAT  section 
conducting  rescue  mission 
for  downed  helo.  He  was 
machinegun  team  leader  at 
the  time.  At  intersection, 
squad  of  Marines  at  each 
corner,  car  came  speeding 
toward  intersection,  Marines 
told  it  to  stop,  fired  warning 

Provide 
security  for 
QRF  that 

was 

assisting  a 
downed 

helo 

Shoot  at  guys 
in  the 

speeding  car 
or  don't  shoot 

Vehicle  not 
reacting  to 
escalation  of 

force 

measures 

(e.g.  warning 
shots  fired) 
may  be  a 
threat 

vehicle  coming 
very  fast,  Marines 
trying  to  get  car  to 
stop  through 
escalation  of  force 
(telling  them  to 
stop,  firing  warning 
shots),  vehicle 
passed  trigger  line 

car  might  be  a 
VBIED 

had  gunners 
engage  vehicle; 
another  decision 

-  don't  have 

gunners  engage 

vehicle 

had 

gunners 

engage 

Stopped  car 
through  force, 
found  3 
military  aged 
males  with 
weapons  and 
grenades  in 

car. 

concerned 

about  welfare  of 
whole  platoon; 
tired  and  hot; 
civilians  in  city 
could  be 

harmed 
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Set  up  in  blocking  position 
on  back  side  of  cemetery  on 
outskirts  of  city,  set  up  in  L- 
shape  blocking  position, 
makeshift  control  measures 
set  up.  A  dump  truck  came 
up  and  wasn't  trying  to  plow 
through,  but  did  cross  control 

measures 

Prevent 

weapons 
and  key 
leader 

movement 

on 

outskirts  of 
city 

shoot  truck 

driver  or  don't 

shoot 

water  trucks 

had  been 

used  as 
VBIEDs,  but 
truck  driver 

did  not  meet 

the  criteria 
for  seeming 
like  a  threat 

truck  driver  was 
old,  by  himself, 
was  not  moving 
really  fast,  in 
outskirts  of  city, 
stopped  when  a 
warning  shot  was 
fired 

probably  not  a 
threat 

did  not  shoot; 
could  have  shot 

him 

did  not 

shoot 

guy  was  just 
lost  and  old 

and  did  not 
realize  where 
he  was; 
Marines 
helped  him  get 
c-wire  out  of 

the  tires  and 
point  him  in 
the  right 
direction 

very  fatigued; 
hot  outside; 
being  shot  at 
from  an 

exclusion  zone 

that  Marines 

could  not  fire 

back  -  morale 
down;  battalion 
let  them  down  - 
went  2  days 
without  water 

Pit  Sgt  in  charge  of  28 
Marines  -  protective  detail  for 
many  important  people. 
Coming  down  unknown  route 
w/o  any  important  people  at 
the  time,  with  no  problems. 

In  first  vie  -  Come  up  on  flat 
open  area,  right  side  of  truck 
sink  into  earth  -  very  fine 
sand  and  water.  At  one  point 
4  of  5  vehicles  stuck  in  mud. 
Had  to  get  out  and  dig 
vehicles  out  -  took  about  5 

hours 

Transport 

and 

protect 

HVI 

between  2 

locations 

Dig  vehicles 
out 

themselves  or 
wait  for  help 

Knew  they 
were  out  in 
the  open  in  a 
bad  part  of 
town 

multiple  vehicles 
sank  slightly  into 
ground 

they  needed  to 
get  the  group  out 
asap 

decided  to  dig 
out  vehicles; 
another  decision 
-  wait  for 
recovery  team  to 
come  out  and 
help 

it  took  5  hours 
to  dig  all 
vehicles  out, 
but  everyone 
made  it  out 
safely 

Environment  - 
out  in  open  in 
dangerous 
neighborhood, 
being  yelled  at 
about  timeline 

from  Lt  Col 
even  though 
mission  had 
priority 
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At  govt  center  and  there  was 
large  protest,  had  a  bunch  of 
state  dept  civilians,  had  to 
get  them  from  downtown  to 
the  Camp.  People  started  to 
throw  rocks  at  state  dept 
people. 

Protect 

and 

transport 
state  dept 
civilians  to 
get  them 
back  to  the 

FOB 

How  to 
disperse  the 
crowd 

a  few  blocks 
away  from 
high  profile 
instances  of 

extreme 

violence 

People  in  crowd 
were  getting 
increasingly  hostile 
-  began  throwing 
rocks 

might  try  to  hurt 
the  state  dept 
civilians  -  need 
to  get  them  out, 
need  crowd  to 
disperse 

Shot  fires  into 
the  ground 

got  everyone 
out  safely 
without  hurting 
anyone 

being 

outnumbered; 
not  being  able 
to  trust  local 
populace,  even 
Iraqi  police; 
being  inside  the 
Iraqi  govt 
center;  knowing 
what  crowd  was 
capable  of 

Told  another  unit  that  it  was 
alright  to  fire  weapons  over 
the  heads  of  his  squad. 
Effects  were  very  short  and 
ended  up  hitting  many 
Marines  in  his  squad,  had  to 
suddenly  deal  with  mass 
casualties  (8-9  Marines 
down) 

CASEVAC 

mass 

casualties 
and  keep 
rest  of 
squad  safe 

How  to  get 
Marines 

CASEVAC 

asap 

Saying 
squad  is  in 
danger  may 
get  air 
assets  to 
support 
quicker  than 
CASEVAC 

many  Marines 
were  badly  hurt 

need  to  get  a  lot 
of  help  very 
quickly 

Said  that  he  was 
in  danger 
instead  of  calling 
for  CASEVAC  to 
get  air  assets  to 
him  faster 

got  everyone 
out 

having  so  many 
casualties  on 
his  team;  low 

on  ammo 

Had  less  than  a  full  squad 
after  taking  casualties  -  had 

2  fire  teams  and  they  were 
taking  mortars  fire  from  an 
enemy  unit  larger  than  them 

initiate  fire  or 

break  contact 

accuracy  of 
mortars  not 
great  -  knew 
effective 

distance  of 

enemy 

mortars 

distance  between 
enemy  and 

Marines 

Marines  were 

within  effective 

distance  of 

mortars 

moved  Marines 
back  and  fought 
back 

Marines  stood 
their  ground 

Marines  were 
upset  about  the 
casualties  they 
took;  low  on 

ammo 
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